---
title: Shortlisting Model for Domain Classification
url: https://www.emergentmind.com/topics/shortlisting-model-slm
type: topic
---

# Shortlisting Model for Domain Classification

Searching arXiv for the referenced paper and closely related context.
A **Shortlisting Model (SLM)**, in the sense of "A Scalable Neural Shortlisting-Reranking Approach for Large-Scale Domain Classification in Natural Language Understanding," is the first-stage model in a two-stage domain classification pipeline for intelligent personal digital assistants (IPDAs) [1804.08064]. In that work, the SLM is exactly the component called **Shortlister**: a fast, lightweight, shared neural network that takes a user utterance, scores all candidate domains, and returns a small top-\(k\) list for downstream reranking. Its role is to make large-scale domain classification feasible when the system must choose among **1,500 domains**, many of them overlapping in functionality, independently evolving, and heterogeneous in quality and popularity [1804.08064].

## 1. Problem setting and two-stage formulation

In the large-scale IPDA setting, an utterance such as “play michael jackson” or “how can I bake an apple pie?” must be mapped to a **domain**, where a domain represents an application or function such as music, recipes, taxi, weather, or smart home [1804.08064]. The paper contrasts this with traditional spoken language understanding deployments containing roughly 20 carefully designed, non-overlapping domains with a shared schema. The large-scale setting instead contains **1,500 overlapping domains**, including many user-developed skills, and the task is to choose the best domain among \(n\) candidates with \(n \approx 1500\) [1804.08064].

The need for shortlisting is driven by latency, memory, compute, and scalability constraints. A naive design would run each domain’s binary domain classifier, multi-class intent classifier, and sequence slot tagger, then rank the resulting hypotheses. The paper states that this is feasible for 8–9 domains, but not for thousands. Running full NLU for every domain is too slow for a production IPDA, requires prohibitive memory footprint and machine count, and does not scale gracefully as new domains appear [1804.08064].

The proposed solution is a **two-stage pipeline**. The first stage is the **Shortlisting Model (Shortlister / SLM)**, which uses only character- and word-level features from the utterance, runs a single shared neural model, scores all domains, and emits a \(k\)-best candidate list. The second stage is **HypRank**, a list-wise reranker that uses richer hypothesis vectors including the Shortlister score, domain-intent-slot NLU scores, user preference signals, and domain popularity or quality signals [1804.08064]. Formally, the system is described as:
$$
\text{Utterance } x \xrightarrow{\text{Shortlister (SLM)}} \text{Top-}k \text{ domains } \{d_1,\dots,d_k\}
\xrightarrow{\text{HypRank}} \hat{d}.
$$

This division of labor is central. The SLM is intentionally lightweight and context-free, optimized for high recall at low latency, while the reranker is feature-rich but invoked only on a small candidate set. A plausible implication is that the SLM should be judged less by 1-best accuracy than by whether it preserves the true domain in the shortlist.

## 2. Architecture of the Shortlisting Model

Shortlister is a **shared neural classifier** with three layers: an orthography-sensitive character+word embedding layer, a **BiLSTM** utterance encoder, and an output layer that scores domains [1804.08064].

Let \(\mathcal{C}\) be the set of characters, \(\mathcal{W}\) the set of words, and \(\oplus\) denote vector concatenation. Character embeddings are
$$
e_c \in \mathbb{R}^{25}, \quad c \in \mathcal{C},
$$
and word embeddings are
$$
e_w \in \mathbb{R}^{100}, \quad w \in \mathcal{W}.
$$
For each word \(w_i\), the model runs forward and backward character LSTMs over its characters and forms an **orthography-sensitive word embedding** \(v_i \in \mathbb{R}^{150}\):
$$
v_i = f^{\mathcal{C}}_{|w_i|} \oplus b^{\mathcal{C}}_1 \oplus e_{w_i}.
$$
This representation concatenates the last forward character-LSTM state, the first backward character-LSTM state, and the word embedding. The paper states that this enables the model to capture subword patterns such as misspellings, morphology, and brands, which are important for domain discrimination [1804.08064].

Given the word-vector sequence \((v_1,\dots,v_m)\), a word-level BiLSTM encodes the utterance. The final forward and backward hidden states are concatenated into the utterance representation
$$
h = f^{\mathcal{W}}_m \oplus b^{\mathcal{W}}_1 \in \mathbb{R}^{200}.
$$
This single vector summarizes the utterance for domain scoring [1804.08064].

The output layer has two variants. In the **global softmax** variant, the model computes
$$
z = Wh + b \in \mathbb{R}^{n}, \qquad o = \text{softmax}(Wh+b),
$$
with
$$
p(d=i \mid x) = o_i = \frac{\exp(z_i)}{\sum_{j=1}^{n}\exp(z_j)}.
$$
In the **per-domain in/out softmax** variant, each domain \(i\) has its own binary classifier:
$$
o^i = \text{softmax}(W^i h + b^i),
$$
where \(o^i_1\) is the in-domain probability and \(o^i_2\) is the out-of-domain probability. At inference time, \(o^i_1\) is used as the shortlisting score, and the top-\(k\) domains are selected accordingly [1804.08064].

The per-domain design is intended to avoid the extreme “winner-takes-all” behavior of a flat softmax in highly overlapping label spaces. This suggests that score independence matters when multiple domains are semantically similar and all deserve to survive into a reranking stage.

## 3. Training objective and optimization

The training objective depends on the output variant. For the global softmax model, the loss is standard cross-entropy:
$$
\mathcal{L}_a = - \sum_{i=1}^{n} l_i \log o_i,
$$
where \(l\) is the one-hot ground-truth domain vector and \(o\) is the softmax output [1804.08064].

For the per-domain in/out softmax model, the loss is
$$
\mathcal{L}_b = -\sum_{i=1}^{n}
\left\{
l_i \log o^i_1 + \frac{1-l_i}{n-1}\log o^i_2
\right\}.
$$
The factor \(\frac{1}{n-1}\) balances the contribution of the many negative domains against the single positive domain, so that in-domain and out-of-domain probabilities are trained on similar scales [1804.08064].

Implementation details are shared with HypRank. The paper specifies the **DyNet** framework, **Adam** optimization with initial learning rate \(4 \times 10^{-4}\), and **variational dropout** for LSTM layers. Training is described as standard mini-batch SGD, with no special handling for class imbalance beyond the balancing factor in \(\mathcal{L}_b\). The model uses neither hierarchical softmax nor sampled softmax nor negative sampling; with \(n=1500\), full-domain training remains practical [1804.08064].

A notable architectural property is that almost all parameters are **shared across domains**. The character embeddings, word embeddings, and BiLSTM encoder are global; only the final output heads scale with the number of domains. The model therefore performs a **single pass per utterance**, followed by either one large matrix multiplication or \(n\) small binary heads, instead of running 1,500 separate domain-specific NLU stacks [1804.08064].

## 4. Interaction with reranking and the role of \(k\)

The SLM is the first half of a deliberately asymmetric system. Shortlisting uses only raw-text lexical features from the utterance and excludes contextual or user-specific signals. Reranking, by contrast, consumes a **hypothesis vector** for each shortlisted domain that includes the Shortlister confidence \(s^d\), intent classification confidence, slot CRF Viterbi path probability, average and maximum slot-tag confidence, pretrained domain, intent, and slot embeddings, user enablement and recent usage indicators, and domain popularity and quality features [1804.08064].

The paper explicitly states: “For the same reason, this work only uses contextual information in the reranking stage, and the utility of including it in the shortlisting stage is left for future work” [1804.08064]. This constraint is not incidental. The shortlister must be computationally cheap enough to run over all domains, so it excludes intent models, slot models, domain popularity statistics, and personalized context. HypRank can afford to use these richer signals because it runs only on the \(k\) candidates returned by the SLM.

The value of \(k\) is empirically important. The paper reports **k-best classification accuracies** for \(k=1,3,5\) and notes that performance “starts to level off at 5-best list” in the large-scale setting [1804.08064]. Accordingly, the HypRank experiments use **\(k=5\)**. This choice balances two considerations stated in the paper: a higher upper bound for reranking, since the true domain is in the top-5 about 96% of the time in the 1,500-domain setting, and modest reranking cost, since only five hypotheses need expensive downstream processing [1804.08064].

This division also clarifies an important misconception. The SLM is not intended to be a complete domain resolver. Its function is candidate generation with high recall@\(k\), not full contextual resolution.

## 5. Evaluation and empirical behavior

The paper evaluates Shortlister in two settings: a **Traditional IPDA** with 20 built-in, high-quality, non-overlapping domains and more than 4M labeled utterances, and a **Large-Scale IPDA** with 1,500 overlapping domains and more than 6M utterances [1804.08064]. In the large-scale dataset, utterances were originally in strict invocation patterns such as “Ask \(\{TAXI\}\) to \(\{get me a ride\}\)” and were preprocessed to remove the domain invocation string.

The train/dev/test partition is structured so that Shortlister and HypRank do not share training data. For the large-scale setting, the splits are: SL train 5M, SL dev 530K, HR train 830K, HR dev 20K, and test 530K. For the traditional setting, they are: SL train 3M, SL dev 415K, HR train 715K, HR dev 20K, and test 420K [1804.08064]. The non-overlap between Shortlister and HypRank training sets is explicitly used to avoid overfitting HypRank to Shortlister outputs.

The evaluation metric for the SLM is **k-best classification accuracy**, described as essentially **recall@k**:
$$
\text{Recall@k} =
\frac{\#\{\text{test samples where ground truth domain is among top k predicted}\}}
{\#\text{test samples}}.
$$
Latency is discussed qualitatively rather than numerically; the paper does not publish exact per-utterance latency figures [1804.08064].

The reported results are as follows:

| Setting | Variant | 1-best | 3-best | 5-best |
|---|---:|---:|---:|---:|
| Traditional (20 domains) | \(softmax_a\) | 95.58% | 98.45% | 98.81% |
| Traditional (20 domains) | \(softmax_b\) | 95.56% | 98.43% | 98.77% |
| Large-Scale (1,500 domains) | \(softmax_a\) | 81.38% | 92.53% | 95.77% |
| Large-Scale (1,500 domains) | \(softmax_b\) | 81.49% | 92.81% | 95.93% |

In the 1,500-domain setting, Shortlister therefore reaches approximately **96% Recall@5**, establishing a strong upper bound for the reranking stage [1804.08064]. The per-domain in/out variant \(softmax_b\) is slightly stronger than the global softmax, especially in the large-scale setting. The paper attributes this to better behavior with overlapping domains [1804.08064].

The paper also reports downstream context. With \(softmax_b\) and \(k=5\) in the large-scale IPDA setting, **Shortlister alone** achieves **81.49%**, while the best reranker, **\(LSTM^C\)**, reaches **93.83%**, against an upper bound of **95.93%** set by Shortlister’s 5-best recall [1804.08064]. This shows that the SLM’s value lies not merely in standalone classification but in preserving enough candidate quality for a richer list-wise reranker to recover most of the remaining gap.

## 6. Scalability, design significance, and broader implications

The scalability of the SLM follows from several explicit design choices. First, feature extraction is shared across domains. Second, each utterance requires only a single encoder pass. Third, the shortlister uses only character and word features, with **no domain embeddings at shortlisting**, no user context, and no domain popularity or intent-slot machinery [1804.08064]. Fourth, the model does not require hierarchical softmax or approximation tricks at 1,500 domains. The paper characterizes the shortlisting cost as essentially one medium-sized BiLSTM plus a \(1500 \times 200\) linear transform, which is substantially cheaper than running domain-specific NLU for every domain [1804.08064].

Several broader design lessons are explicit or strongly suggested by the study. A two-stage architecture becomes necessary when the label space is very large and overlapping. Shared lexical encoders can provide high recall@\(k\) even under severe scale. Character-level modeling is particularly useful when domain discrimination depends on noisy user text, morphology, or brand and skill names. Designing the first stage around **Recall@k** rather than 1-best accuracy is operationally more important because reranking can only repair errors if the true domain survives the shortlist [1804.08064].

The paper also frames the per-domain binary-softmax variant as a response to overlapping label spaces. A plausible implication is that once the candidate space becomes semantically entangled, independent in/out scoring may be a more natural front-end than a single normalized softmax. At the same time, the work deliberately excludes contextual features from the SLM, leaving open the question of whether context-aware shortlisting can improve recall without violating latency budgets.

Within the scope of [1804.08064], however, the definition is precise. A Shortlisting Model is not a general small language model, a voting procedure, or a feature selector. It is a **single, shared BiLSTM-based text classifier** that encodes an utterance as
$$
h = f^{\mathcal{W}}_m \oplus b^{\mathcal{W}}_1,
$$
scores every domain either by
$$
o = \text{softmax}(Wh+b)
$$
or by
$$
o^i = \text{softmax}(W^i h+b^i), \qquad \text{score}_i=o^i_1,
$$
and returns the top-\(k\) domains for subsequent list-wise reranking [1804.08064]. In the evaluated 1,500-domain IPDA setting, that design yields approximately **95.93%** 5-best accuracy, making large-scale domain classification operationally tractable without running full NLU across the entire domain inventory [1804.08064].

Source: https://www.emergentmind.com/topics/shortlisting-model-slm