Semantic Outlier Removal with Embedding Models and LLMs

Abstract: Modern text processing pipelines demand robust methods to remove extraneous content while preserving a document's core message. Traditional approaches such as HTML boilerplate extraction or keyword filters often fail in multilingual settings and struggle with context-sensitive nuances, whereas LLMs offer improved quality at high computational cost. We introduce SORE (Semantic Outlier Removal), a cost-effective, transparent method that leverages multilingual sentence embeddings and approximate nearest-neighbor search to identify and excise unwanted text segments. By first identifying core content via metadata embedding and then flagging segments that either closely match predefined outlier groups or deviate significantly from the core, SORE achieves near-LLM extraction precision at a fraction of the cost. Experiments on HTML datasets demonstrate that SORE outperforms structural methods and yield high precision in diverse scenarios. Our system is currently deployed in production, processing millions of documents daily across multiple languages while maintaining both efficiency and accuracy. To facilitate reproducibility and further research, we release our implementation and evaluation datasets.

• The paper introduces SORE, an embedding-driven method that identifies core content via metadata and cosine similarity.
• It achieves an F-score of 0.732 and processes documents in ~200ms, cutting costs by up to 25x compared to LLM approaches.
• The method demonstrates robust multilingual performance and transparent outlier detection for scalable semantic extraction.

Semantic Outlier Removal with Embedding Models and LLMs: An Expert Summary

The presented work introduces SORE (Semantic Outlier Removal), an embedding-based approach for cleaning web documents by removing non-essential segments while retaining core content. Unlike traditional HTML-structure heuristics or expensive LLM approaches, SORE leverages multilingual embeddings and approximate nearest neighbor (ANN) search to provide high-precision content extraction with scalability suitable for large-scale production systems.

Core Algorithm and System Architecture

SORE is implemented as a multi-step pipeline, focused on industrial robustness, transparency, and efficiency. The main workflow comprises:

1. Segmentation and Embedding: HTML documents are parsed into structural segments (typically sentences or paragraphs), which are embedded using secure, production-ready multilingual embedding models (e.g., AWS Titan, Cohere, or open-source e5-large). Document metadata—especially title and description—serve as the semantic reference anchor.
2. Core Identification: For each segment, cosine similarity to the metadata embedding is calculated. The top-k% of segments most similar to the metadata are labeled as "core content," providing a principled, language-agnostic mechanism for identifying the main theme.
3. Outlier Detection: SORE employs pre-embedded outlier groups typifying boilerplate and unwanted content (e.g., ads, legal disclaimers, navigation). Each non-core segment is checked both for semantic proximity to these outliers and for semantic remoteness from core content. Tunable thresholds and ANN search (via HNSW from Voyager) ensure computationally efficient segment scoring.
4. Segment Removal and Transparency: Segments exceeding the distance cutoff or matching outlier groups are excised, with removal rationales explicitly logged. This decision traceability supports debugging and post-processing audits.

The architecture enables batch processing, index preloading, distance optimization (via inner product), and outlier group refinement. SORE’s implementation is optimized (in Java) to process a typical HTML document in ~200ms, mainly dominated by the embedding API latency.

Comparative Evaluation

Comprehensive empirical evaluation is performed on in-house multilingual datasets, SORE-small (manually annotated) and SORE-large (automatically extracted). SORE is benchmarked against both ReadabilityJS (HTML-structural extraction) and multiple LLM-based systems with variant prompting strategies.

Extraction Accuracy [see Table 1 in paper]:

• SORE yields an F-score of 0.732 on SORE-small, achieving ~96% of the best LLM approach (0.765), and surpassing ReadabilityJS (0.678).
• SORE’s precision (0.700) markedly exceeds structural baselines (0.569), though recall is slightly reduced compared to heuristic extractors—reflecting the algorithm’s favoring of precision in cases of uncertainty.
• On SORE-large, SORE tuned with AWS Titan embeddings demonstrates competitive performance at substantially reduced cost.

Cost and Latency [see Table 2 in paper]:

• SORE (with AWS Titan) delivers typical processing in 200ms per document, compared to 2,500ms for LLM-based extraction.
• The cost per million documents: $600 for SORE (AWS Titan),$15,000 for LLMs—a total operational cost reduction of ~25x at scale.
• In production (30M doc/month), SORE yields savings in the order of hundreds of thousands of USD annually, enabling semantic extraction viability for industrial-level workloads.

Parameter Sensitivity and Practical Tuning

Experiments reveal several practical tuning levers:

• Embedding dimensionality: Higher-dimensional models (e.g., 1024d) yield better separation in embedding space.
• Core content proportion (k%): Lower percentages increase selectivity at the risk of recall loss.
• Distance cutoff (d): Higher cutoffs improve precision; lower cutoffs favor recall.
• Embedding model choice: Commercial options (AWS Titan, Cohere) balance speed, scalability, and performance; open-source e5-large offers viable public deployment under licensing constraints.

Notably, SORE exhibits strong multilingual generalization, accurately flagging and excising boilerplate elements across English, Chinese, French, Spanish, and more, with no need for language-specific rule sets. Case analyses further substantiate the approach’s robustness in practical, noisy, and multilingual settings.

Implications and Future Directions

SORE’s embedding-driven methodology articulates a middle ground between traditional heuristic systems—limited by language and structure—and costly, often non-deterministic LLM content extraction. The method’s major practical contributions include:

• Competitive precision and operational reliability in real-world, multilingual document pipelines.
• Transparent, explainable, and deterministic removal mechanisms amenable to systematic refinement.
• Enabling cost-effective deployment at previously unattainable scales for semantic web preprocessing.

The system is further distinguished by its open commitment to public release of both codebase and evaluation datasets, supporting reproducibility and extension.

Theoretical and Applied Prospects:

• The SORE framework can be extended to incorporate domain-adaptive outlier clusters, dynamic retraining on misclassified segments, or sentiment/tone-based segment filtering, making it adaptable for layered, adaptive content cleaning.
• Future research may integrate external knowledge bases or task-specific classifiers to refine outlier definitions dynamically.
• The approach’s transparency and logging are valuable for continuous learning frameworks and for addressing edge cases where over-removal risks missing crucial context.

Conclusions

The SORE system demonstrates that embedding-space semantic similarity, systematically anchored to document metadata and refined by ANN-driven outlier detection, can approach LLM-level extraction accuracy—while unlocking practical industrial scalability. SORE enables semantic, multilingual content cleaning that is both explainable and economically viable, representing an effective architectural template for document preprocessing pipelines in text-heavy AI applications.