A Unified Structured Query Understanding Framework for Industrial Semantic Search

Abstract: Query understanding in large-scale industrial search systems is typically implemented as a cascade of disparate, task-specific components. While individually optimizable, this fragmented architecture incurs high maintenance overhead and results in inconsistent behaviors, particularly for long-tail queries. In this work, we propose and deploy a unified structured query understanding system that consolidates these heterogeneous functions into a single Small LLM (SLM) that performs schema-constrained generation. To address the data bottlenecks inherent in unified modeling, we introduce Query Illuminator, a dual-purpose framework serving as: (i) a teacher model for high-quality auto-annotation and distillation, and (ii) a surrogate judge for scalable evaluation where human labels are scarce. We validate this approach through extensive offline and online tests within LinkedIn's Job Search system. Furthermore, we demonstrate the framework's horizontal extensibility through a cross-domain case study on People Search. The results show improved user engagement and reduced operational costs, achieved while satisfying strict low-latency serving constraints on limited GPU resources.

• The paper unifies routing, tagging, rewriting, facet suggestion, and trust decisions in a schema-constrained 1.5B model, reducing request failures by 17.13% in online testing.
• Multi-task fine-tuning and teacher–student distillation improve location-extraction F1 from 0.885 to 0.924, while the compact student approaches larger teacher models across English, German, Spanish, and French.
• The production framework supports low-latency deployment and cross-domain reuse, delivering up to 75% relative gains in graded relevance for People Search and improving downstream retrieval without retraining embedding models.

Motivation and problem setting

Industrial semantic search systems typically implement query understanding as a cascade of independently trained components—entity taggers, intent classifiers, rewriters, and rule-based triggers. The authors argue that this fragmentation raises maintenance costs, produces inconsistent behavior across query types, and amplifies errors on long-tail queries. The problem is studied in LinkedIn's Semantic Job Search (SJS), which indexes more than $10^7$ documents at over $10^4$ queries per second. Although the median SJS query is only two words long, the tail is heavy: Premium-member SJS queries reach 5 words at the 90th percentile and 18 words at the 99th percentile, versus 11 words for Classical Job Search (CJS). Queries such as "easy apply remote data entry clerk" or "salary > X or remote in US" interleave UI constraints, numerical filters, and geographic reasoning in ways that cascaded pipelines resolve inconsistently.

The paper's central proposal is to replace the fragmented stack with a single Small LLM (SLM) that performs schema-constrained generation of all query-understanding artifacts: a routing decision $r$, structured tags $t$, an optional rewrite $w$, facet triggers, and trust/safety decisions $s$. Routing is generated jointly with the other fields rather than by a separate intent classifier, so inter-field dependencies (e.g., conditioning a rewrite on detected intent) are captured within one forward pass with deterministic latency bounds.

Architecture: online inference and Query Illuminator

The system decouples two competing objectives into asynchronous environments. Online, a 1.5B-parameter model (Qwen2.5-1.5B backbone) performs single-pass structured generation exposed as tool invocation with typed JSON arguments; guided grammar decoding via xgrammar enforces the schema. Offline, the Query Illuminator—an 8B teacher LLM—serves two roles: it generates schema-compliant supervision targets for distillation, and it acts as a policy-grounded surrogate judge comparing candidate outputs against references with rubric-based scores and a structured error taxonomy. Policy specifications, golden sets, and prompts are versioned together, and live traffic is rescored daily to monitor distribution shift.

The router emits four coarse categories: criteria queries (extract constraints), self-referential queries (trigger profile-aware rewriting), unsupported/unclear intents (fall back to keyword-only retrieval), and trust-violating queries (block). This conservative fallback design directly targets over-extraction failures of legacy pipelines.

Training methodology

Training uses roughly 14K production queries annotated through human-in-the-loop bootstrapping: Illuminator-generated candidates reviewed by expert annotators. Because historical keyword-centric logs do not reflect semantic-query traffic—a chicken-and-egg problem the authors acknowledge explicitly—the dataset is augmented via GPT-based rewriting of keyword queries into natural-language forms with randomized tool invocations, plus Alpaca-style synthetic instruction data with injected company names and locations. A continuous active-learning loop prioritizes samples with high teacher–student disagreement.

The training recipe combines multi-task supervised fine-tuning with knowledge distillation, using a loss $$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{SFT}} + \lambda \mathcal{L}_{\text{KD}}$$ where $\mathcal{L}_{\text{KD}}$ is KL divergence between teacher and student output distributions. Two ablations support the design choices:

Setting	Precision	Recall	F1
Location tool only	0.899	0.871	0.885
Multi-task (all tools)	0.928	0.919	0.924

Multi-task fine-tuning lifts location-extraction F1 from 0.885 to 0.924, attributed to positive transfer and regularization against lexical overfitting. However, GLUE diagnostics reveal a trade-off resembling an alignment tax: gains on semantic similarity tasks but regressions on natural language inference, suggesting objective interference. For facet suggestion, the hybrid SFT + Distill strategy achieves the lowest evaluation loss (0.3428) and matches the teacher's F1 of 0.81, closing most of the student–teacher gap.

Serving infrastructure

The serving stack wraps vLLM behind a hosted LLM service with a cache layer. Three optimizations are notable. Guided grammar decoding exploits the fixed tool set. Client-side batching of point-wise facet scoring reduces p95 latency by over 70% relative to server-side batching at equal QPS. Prefix caching exploits the shared system prompt and per-session member profile. On a single NVIDIA A100 MIG-4g.40G GPU, query tagging sustains median end-to-end latency below 300 ms up to 15 QPS (P99 ≈ 626 ms), while facet suggestion holds ~110 ms median latency but degrades earlier at P99 due to combinatorial expansion over 20–30 candidates. These figures demonstrate feasibility under constrained GPU budgets, though the facet-suggestion P99 degradation at high load indicates limited headroom.

Offline evaluation results

Multilingual tagging evaluation across English, German, Spanish, and French yields two findings. First, scaling is not monotonic: the 7B model improves English recall (0.892 vs. 0.843) but slightly reduces precision compared to the 1.5B baseline. Second, the distilled 1.5B student recovers much of the teacher's quality—for example, German recall of 0.956 versus the 7B teacher's 0.949—at a far smaller serving footprint. Human evaluation of company extraction shows the Illuminator teacher reaching 91.50% simple accuracy and 90.17% F-measure, exceeding both the legacy baseline (84.00% / 79.22%) and a GPT-based baseline (90.50% / 89.84%), though with higher false-alarm rate than the legacy system (11.02% vs. 7.76%).

Surrogate judge validation

Because subtle failure modes (hallucinated filters, incorrect facet typing) resist automatic metrics, the Illuminator doubles as an LLM judge emitting rubric scores for facet extraction and query hygiene, plus a mistake taxonomy. Validated on 4,903 human-labeled people-search queries with ~300K internal samples, judge–human agreement reaches Cohen's $\kappa = 0.77$ against a human-expert ceiling of 0.83, and JSON parse success ranges from 99.96% to 100%. The authors are appropriately cautious: judge scores are treated as auxiliary development signals rather than ground truth, and rigorous quantification of judge–human correlation is deferred to future work.

Online A/B results

Large-scale randomized experiments on Semantic Job Search and Job Alerts (ramped to a 50/50 split over 14 days, two-sided $t$-tests at $10^4$0) show statistically significant improvements:

Surface	Metric	Relative change
Job Search	Request failures	−17.13%
Job Search	Search sessions	+0.74%
Job Search	Successful apply or save	+0.49%
Job Search	Apply-to-viewport ratio	+0.14%
Job Alert	Total job applications	+0.23%
Job Alert	Searches without results	−0.65%

The 17.13% reduction in request failures—driven mainly by routing and tagging improvements—is the strongest result, since failed requests prevent retrieval entirely. Engagement gains are positive but modest in magnitude, consistent with incremental relevance improvements layered on an already-tuned system.

Cross-domain extensibility and downstream consumption

Porting the framework to Semantic People Search (>10⁹ documents) required only redefining the output schema and reusing the serving stack, with templated synthetic queries from a knowledge graph compensating for sparse early ramp logs. In a U.S. ramp, 70.1% of traffic routes to the semantic path, 3.7% falls back safely, and <0.1% is trust-blocked. Offline graded-relevance replay shows large relative lifts when retrieval consumes decoded structured queries: +57% on navigational and +75% on exploratory queries. Separately, injecting tagged company names into an existing embedding-based retrieval (EBR) prompt—without any EBR retraining—improves top-50 metrics by +1.8% precision, +2.2% recall, and +1.7% NDCG, illustrating that structured outputs are consumable by downstream models without architectural change.

Limitations and open questions

The paper concedes several boundaries plainly. Context selection is heuristic: curated high-density features and router gating substitute for principled learning-based evidence selection, leaving latency/hallucination trade-offs unresolved. Short, highly ambiguous queries impose an upper bound on non-interactive resolution; interactive disambiguation is out of scope. Portability claims are empirical—the People Search case study demonstrates reuse, but broader cross-vertical validation remains untested. Surrogate-judge metrics risk drift if prompts diverge from evolving product policy, and full correlation analysis with human annotations is pending. Finally, the appendix's Lagrangian argument for multi-task fine-tuning as capability retention relies on convexity assumptions that deep networks do not satisfy; it is offered as interpretation, not proof.

Conclusion

This paper presents a production-deployed framework that consolidates routing, tagging, rewriting, facet suggestion, and trust handling into a single schema-constrained 1.5B SLM, supported by a dual-purpose offline teacher/judge (Query Illuminator). Deployment evidence includes a 17.13% reduction in request failures, consistent engagement gains, multilingual offline quality approaching larger teachers, and demonstrated portability to a second domain with substantial graded-relevance lifts. The main open questions concern learning-based context selection, quantified judge validation, and generalization beyond the two verticals studied.