Luxical: Efficient Lexical-Dense Text Embeddings

• The paper presents a hybrid embedding architecture that combines sparse TF–IDF inputs and a compact MLP to produce dense text embeddings at high throughput.
• It leverages knowledge distillation using a Gram-matrix KL-divergence loss to align its output with state-of-the-art transformer models.
• Empirical results show Luxical achieving up to 60× CPU speedups over transformer baselines while retaining competitive retrieval accuracy.

Luxical is a high-throughput text embedding framework designed to bridge the computational efficiency of classical lexical models (e.g., TF–IDF, FastText) and the representational flexibility of transformer-based dense models. Its architecture and training regimen are targeted specifically at large-scale text organization workflows, such as document deduplication, clustering, retrieval, and LLM data curation in web-scale corpora. Luxical achieves this by combining sparse TF–IDF representations, a compact feed-forward neural network, and knowledge-distillation from state-of-the-art transformer embedding models, yielding dense vector outputs at operational costs comparable to classic lexical approaches while matching the quality of neural baselines in coarse-grained applications (DatologyAI et al., 9 Dec 2025).

1. Lexical–Dense Embedding Architecture

Luxical constructs document embeddings via a pipeline that starts with a sparse TF–IDF vector representing token and $n$-gram occurrence, followed by projection through a shallow, high-throughput multilayer perceptron (MLP) with ReLU non-linearities and intervening $\ell_2$-normalization layers. This design yields a dense output embedding (e.g., 192 dimensions) with unit norm, optimized for efficient similarity computation and stable numerical properties (DatologyAI et al., 9 Dec 2025).

TF–IDF Sparse Input Construction:

Given input document $d$ drawn from a vocabulary of $V$ tokens (including $n$-grams), the term frequency and inverse document frequency are computed as:

• $\mathrm{tf}_{i,d} = n_{i,d}$ (raw count)
• $\mathrm{idf}_i = \log(N / n_i)$, with $N$ the total corpus size and $n_i$ the count of documents containing term $i$.
• The unnormalized vector is $$\tilde{s}_i = \mathrm{tf}_{i,d} \cdot \mathrm{idf}_i$$, normalized as $$\mathbf{s} = \tilde{\mathbf{s}}/\|\tilde{\mathbf{s}}\|_2$$.

Sparse-to-Dense Network Structure:

The canonical "luxical-one" configuration is:

• Input layer: $V \approx 2\times10^6$ dimensions (sparse)
• Layer 1: $2\mathrm{M} \rightarrow 92$ (Linear $\rightarrow$ ReLU $\rightarrow$ $\ell_2$-norm)
• Layer 2: $92 \rightarrow 3{,}072$ (Linear $\rightarrow$ ReLU $\rightarrow$ $\ell_2$-norm)
• Layer 3: $3{,}072 \rightarrow 3{,}072$ (Linear $\rightarrow$ ReLU $\rightarrow$ $\ell_2$-norm)
• Output layer: $3{,}072 \rightarrow 192$ (Linear $\rightarrow$ $\ell_2$-norm)

Efficient sparse-by-dense projection exploits the sparsity of TF–IDF: $$A\mathbf{s} = \sum_{i \in \mathrm{nz}(\mathbf{s})} s_i \mathbf{a}_i$$, making the first layer computationally trivial relative to tokenization.

2. Knowledge Distillation and Training Regimen

Luxical leverages knowledge distillation to align its embedding geometry with that of a much larger dense transformer model (e.g., arctic-embed-m-v2.0, 256-dim). Training proceeds by minimizing a Gram-matrix KL-divergence loss between batches of normalized student (Luxical) embeddings $S \in \mathbb{R}^{n \times d}$ and teacher embeddings $T \in \mathbb{R}^{n \times d_t}$: $$G_s = SS^\top,\ G_t = TT^\top,\ \mathcal{L}_\mathrm{distill} = \tau^2 \mathrm{KL}(\mathrm{softmax}(\widehat{G}_s/\tau) \parallel \mathrm{softmax}(\widehat{G}_t/\tau))$$ where the diagonals are removed to ensure the loss compares only mutual similarities. Training is conducted over 50 million FineWeb documents, batch size 3,072, with Adam optimizer and temperature $\tau=3.0$, entirely on CPU hardware. No auxiliary regularizers are used.

3. Empirical Performance and Speed–Quality Trade-Offs

Throughput

Luxical demonstrates superior throughput versus transformer-based and FastText baselines:

Model	CPU Only (docs/s)	GPU Accelerated (docs/s)
Luxical-one	6,000	–
all-MiniLM-L6-v2	90	700
Qwen3-0.6B embedding	5	200

Relative CPU speedups reach 60× over MiniLM and 1,200× over Qwen3. Luxical's throughput exceeds MiniLM on GPU by ≈8.6× (DatologyAI et al., 9 Dec 2025).

Retrieval and Curation Accuracy

Model	Top-1 Acc.	Top-10 Acc.	Top-100 Acc.
Mxbai-L-v1	85%	95%	99%
Arctic-2.0-M	82%	92%	98%
LEAF-MT	75%	88%	96%
Luxical-one	70%	85%	95%
all-MiniLM-L6-v2	55%	70%	90%

In strict top-1 document-half matching retrieval, Luxical trails leading transformers by 10–15 points. However, by top-10, over 70% of the quality gap is mitigated, making Luxical suitable for coarse-grained deduplication or clustering tasks (DatologyAI et al., 9 Dec 2025).

In large-scale data curation for LM training, Luxical matches both FastText and heavy transformer classifiers in downstream accuracy after filtering (36.4% avg. zero-shot acc. across benchmarks), while running at >10× higher throughput than BERT models (DatologyAI et al., 9 Dec 2025).

4. Comparison with Lexical-Dense Paradigms

Luxical belongs to a broad category of dense lexical and lexical-dense hybrid methods developed to fuse the speed and mutual exclusivity of sparse lexical approaches with the coverage and flexibility of dense semantic encoders.

• CluSD combines initial sparse retrieval (e.g., SPLADE/BM25-T5), followed by cluster-based selection of candidate blocks for dense retrieval, using a two-stage process (feature overlap pruning, then LSTM-guided refinement). Dense and sparse scores are fused by linear interpolation. CluSD achieves near-oracle dense retrieval accuracy while evaluating only ≈0.2–0.5% of all embeddings, circumventing the full cost of dense retrieval (Yang et al., 15 Feb 2025).
• DLRs and DHRs: Techniques such as Densified Lexical Representations (DLR) and Dense Hybrid Representations (DHR) densify high-dimensional term-weight vectors via max-pooling over slices, enabling direct inner-product scoring. DHRs further concatenate DLRs with dense semantic vectors in a gated fashion, supporting unified, fast, and accurate retrieval frameworks without CPU–GPU orchestration overhead (Lin et al., 2022).
• DENSIFIER projects generic pretrained word embeddings onto ultra-dense orthogonal subspaces tailored for lexical properties, producing substantial inference speedups with minor or no loss in task quality. Empirical results indicate 10–50× throughput improvements, with only ≈1–2% absolute accuracy drop at extreme compression (e.g., 400→4 dim) (Rothe et al., 2016).

5. Use Cases, Limitations, and Recommended Practice

Use Cases:

• Large-scale document clustering, deduplication, and filtering pipelines where absolute top-1 precision is less critical than throughput and bulk similarity structure (e.g., semantic deduplication, PII filtering, training data curation).
• Serving as a backend for first-stage embedding on commodity CPU prior to light-weight application-specific heads (e.g., nearest-neighbor retrieval, shallow classifiers).
• Scenarios where repeated downstream access justifies the up-front cost of embedding all documents once.

Limitations:

• Luxical is sub-optimal for short or fine-grained queries, or for applications requiring high-precision, nuanced, or reasoning-driven ranking. Here, full transformer encoder models are still unmatched (DatologyAI et al., 9 Dec 2025).
• All evaluations have focused on batch and large-corpus scenarios; single-sample, real-time use-cases may see a reduced relative speedup due to batch amortization.

Production Integration:

• Recommended as part of a Python+Rust+Numba throughput stack, embedding all documents in advance for batch-oriented text organization.
• Dropout or layer normalization are omitted for maximal embedding speed; further ablations regarding width, depth, or final dimension can be tuned to target specific speed–quality tradeoffs.

6. Placement in the Landscape of Embedding Methods

Luxical exemplifies a design shift toward leveraging the regularities captured by powerful teacher models, coupled with the simplicity and speed of lexical features and compact neural architectures, as evidenced by similar progressions in DHR, CluSD, and DENSIFIER paradigms (Yang et al., 15 Feb 2025, Lin et al., 2022, Rothe et al., 2016). By focusing on efficient sparse-by-dense operations and unit-normalized layers, it maximizes hardware utilization for streaming, batch, or weakly supervised learning regimes.

A plausible implication is that as LLM training and large-scale text engineering further specialize, adoption of lexical-dense architectures such as Luxical and its conceptual analogues will become standard in workflows prioritizing cost, speed, and bulk semantic organization over absolute precision in fine-grained modeling tasks.

7. Open Source Availability and Future Directions

Luxical is released as open-source software, with model checkpoints ("luxical-one") and tokenizer implementations (Rust/Arrow, Numba/CPU) available for immediate integration (DatologyAI et al., 9 Dec 2025). Ongoing and future directions include ablations for model width and depth, deployment in hybrid two-stage pipelines (mirroring CluSD), and exploration of further teacher architectures for broadening representational coverage within the established high-efficiency framework.

The cumulative research body demonstrates that integration of lexical, dense, and hybrid embedding strategies is a dominant trend for scalable, robust, and hardware-efficient text representation. Systems such as Luxical, DHR, CluSD, and DENSIFIER collectively define the contours of this evolving approach to language engineering at scale (DatologyAI et al., 9 Dec 2025, Lin et al., 2022, Yang et al., 15 Feb 2025, Rothe et al., 2016).