Luxical: High-Speed Lexical-Dense Text Embeddings (2512.09015v2)

Abstract: Frontier LLM quality increasingly hinges on our ability to organize web-scale text corpora for training. Today's dominant tools trade off speed and flexibility: lexical classifiers (e.g., FastText) are fast but limited to producing classification output scores, while the vector-valued outputs of transformer text embedding models flexibly support numerous workflows (e.g., clustering, classification, and retrieval) but are computationally expensive to produce. We introduce Luxical, a library for high-speed "lexical-dense" text embeddings that aims to recover the best properties of both approaches for web-scale text organization. Luxical combines sparse TF--IDF features, a small ReLU network, and a knowledge distillation training regimen to approximate large transformer embedding models at a fraction of their operational cost. In this technical report, we describe the Luxical architecture and training objective and evaluate a concrete Luxical model in two disparate applications: a targeted webcrawl document retrieval test and an end-to-end LLM data curation task grounded in text classification. In these tasks we demonstrate speedups ranging from 3x to 100x over varying-sized neural baselines, and comparable to FastText model inference during the data curation task. On these evaluations, the tested Luxical model illustrates favorable compute/quality trade-offs for large-scale text organization, matching the quality of neural baselines. Luxical is available as open-source software at https://github.com/datologyai/luxical.

• The paper introduces a hybrid model combining TF-IDF ngram features and a compact neural projection to deliver dense embeddings with 3x–100x throughput improvements.
• The paper employs a contrastive distillation loss that aligns sparse lexical representations with transformer teacher outputs for competitive semantic retrieval.
• The paper demonstrates efficient document organization, achieving transformer-level clustering quality at a fraction of the computational cost.

Luxical: High-Speed Lexical-Dense Text Embeddings

Introduction

The demand for organizing and filtering web-scale corpora for LLM pretraining has exposed fundamental trade-offs in the current embedding infrastructure. Lexical classifiers (e.g., FastText) offer extremely high throughput but are limited to producing score-based outputs, which curtails their expressivity in retrieval and clustering workflows. Conversely, transformer-based encoders provide high-quality, dense vector embeddings but incur prohibitive runtime costs when deployed as primary embedding stages in trillion-token pipelines. "Luxical: High-Speed Lexical-Dense Text Embeddings" (2512.09015) introduces Luxical, a hybrid approach that synthesizes lexical features (via TF-IDF ngram frequencies) and dense neural projections to realize a model that approaches transformer-level semantic expressivity while achieving 3x–100x higher computational throughput.

Model Architecture

The defining characteristic of Luxical is its sparse-to-dense embedding pipeline. Input tokens are converted into a bag-of-ngrams TF representation, weighted by IDF, and normalized to form a high-dimensional sparse vector. This vector is mapped to a dense, normalized embedding by a compact multi-layer ReLU MLP—the architectural size is small enough to ensure CPU-friendly inference but expressive enough to capture meaningful semantic relationships.

Figure 1: Sparse-to-dense architecture of Luxical. A sparse vector of normalized ngram frequencies (the TF-IDF vector) is projected through a small MLP to produce a dense embedding.

Sparse-by-dense multiplication, efficiently implemented via Numba, exploits the inherent sparsity of document ngram statistics to minimize memory bandwidth and compute, shifting performance bottlenecks to tokenization rather than projection. The pipeline relies on an Arrow-based Rust tokenizer, circumventing Python garbage collection overhead that typically plagues high-throughput text processing.

Distillation Objective

For training, Luxical adopts a contrastive embedding distillation framework. Batches of student and teacher embeddings are mapped to their respective Gram matrices (off-diagonal), and the Kullback–Leibler divergence between temperature-scaled, diagonal-stripped Gram matrices is minimized. This aligns the embedding space of the student with the similarity manifold induced by a larger, high-fidelity transformer teacher.

Figure 2: Contrastive distillation loss recommended when training Luxical models.

The instantiation detailed in the paper uses a two million 5-gram vocabulary determined by the Space-Saving algorithm and a 192-dimensional output embedding. The teacher model is Arctic-Embed-2.0-M ("Arctic-Embed: Scalable, Efficient, and Accurate Text Embedding Models" (Merrick et al., 8 May 2024, Yu et al., 3 Dec 2024)), which offers a strong trade-off between throughput and embedding quality.

Empirical Evaluation

Throughput Analysis

Comprehensive end-to-end throughput measurements reveal that Luxical achieves substantial speed gains compared to MiniLM-L6-v2 and especially Qwen3-0.6B, both on CPU and when the latter are GPU-accelerated. Measured on 100,000 FineWeb documents, Luxical outpaces MiniLM by factors exceeding 3x, and eclipses Qwen3 by two orders of magnitude.

Figure 3: End-to-end throughput (web documents per second) when embedding 100,000 documents with Luxical and transformer baselines.

Symmetrical Retrieval: Document-Half Matching

For the document-half matching task, which evaluates symmetric semantic alignment by identifying matching halves from split documents, Luxical yields retrieval error rates comparable to or better than MiniLM-L6-v2, especially at larger retrieval window sizes. The error curve approaches that of the teacher model (Arctic-2.0-M) and advanced distilled baselines (LEAF-MT, Mxbai-L-v1) for high-recall, document-level retrieval—ideal for clustering and deduplication where top-k recall is more critical than strict top-1 precision.

Figure 4: Document-half matching error rates as a function of retrieval window size for Luxical and transformer baselines.

Classifier-Based Filtering for LM Data Curation

The throughput of the pipeline deploying a shallow MLP classifier over frozen Luxical embeddings matches FastText-based DCLM scoring and far exceeds transformer baselines. Downstream, LMs trained on filtered subsets curated by Luxical-based scoring attain accuracy on MMLU and other benchmarks equivalent to transformer-filtered data and clearly superior (by a substantial margin) to random selection.

Figure 5: Classification throughput.

These results confirm that Luxical occupies an optimal Pareto frontier for workflows prioritizing high-throughput embedding with preservation of coarse semantic similarity.

Discussion and Theoretical Implications

Luxical operationalizes the insight that high-throughput, bag-of-ngram lexical statistics, when coupled with weakly-supervised neural distillation, suffice for a broad range of web-scale data organization tasks. By eschewing per-token contextualization inherent in transformer layers and learning to map sparse lexical structures into semantically shaped dense spaces, Luxical exposes a new operating point: it is neither strictly lexical nor fully contextual, but achieves the critical recall and clustering performance needed for most data curation, deduplication, and filtering pipelines in LLM pretraining.

However, the architecture deliberately sacrifices fine-grained semantic distinction and reasoning acuity required in high-precision retrieval or multi-hop question answering. This choice reflects the intended deployment for symmetric similarity and batch-level organization rather than direct search or NLU. The generalizability of the methodology to multilingual corpora, alternate tokenizations, or more complex sparse feature spaces is an open direction for future exploration.

Practical Impacts and Prospective Developments

Adoption of Luxical or analogous hybrid models can dramatically reduce the CPU-hours necessary for initial document representation, enabling practitioners to iteratively develop filtering policies and embedding-driven data slicing. The methodology is apt for integration into multi-stage pipeline architectures; for instance, Luxical embeddings can serve as the first-stage coarse filter before more expensive transformer refinement. The negligible runtime overhead of downstream MLP classifiers further suggests that embedding once with Luxical is amortized across many subsequent analytic tasks.

Given the open-source codebase and pretrained models, as well as the modular nature of the distillation framework, it is plausible that future work will involve distilled, multilingual, or further compressed Luxical-like variants, as well as more systematic evaluations across diverse hardware and correlation with downstream LM utility for both symmetric and asymmetric tasks.

Conclusion

Luxical demonstrates that standard high-dimensional lexical statistics combined with efficient neural projection and distillation from large transformer teachers can yield embeddings with throughput and flexibility unattainable by either class of models alone. The numerical results establish that for web-scale document organization, clustering, and classifier-based filtering, Luxical attains quality on par with smaller transformers at drastically reduced computational cost. This has direct implications for corpus curation workflows and presents Luxical as a robust, practical point on the compute/quality frontier for web-scale text processing (2512.09015).