SemHash-LLM: A Multi-Granularity Semantic Hashing Framework for Document Deduplication

Abstract: Large scale document deduplication must preserve semantic equivalence while remaining efficient over massive corpora. We present SemHash LLM, a multi granularity framework that unifies semantic projection hashing, attention weighted MinHash, contrastive boundary learning, and selective LLM based adjudication. The method combines character, token, and document level signals through gated fusion, then applies a cascaded filtering pipeline for efficient candidate reduction. Semantic projection hashing learns compact binary codes in distilled LLM embedding space, while attention weighted Min- Hash suppresses boilerplate and emphasizes informative content. Adaptive decision boundaries and uncertainty estimation further improve robustness across template pollution, short text perturbation, containment, and viral fragments. Experiments show that SemHash LLM achieves strong duplicate detection quality with less than one percent neural verification cost.

• The paper proposes a unified deduplication framework integrating semantic projection hashing, attention-weighted MinHash, and contrastive boundary learning to enhance precision and efficiency.
• It employs multi-granularity feature extraction across character, token, and semantic levels, reducing neural verification to under 1% of candidate pairs.
• Empirical results on a 100GB corpus demonstrate a composite final score of 91.05, significantly outperforming traditional deduplication methods.

SemHash-LLM: Multi-Granularity Semantic Hashing for Large-Scale Document Deduplication

Problem Motivation and Limitations of Prior Art

Large-scale document deduplication is fundamental to the integrity and efficiency of web-scale LLM pretraining, data pipelines, and retrieval systems. Notably, traditional deduplication methods such as MinHash and SimHash are limited to surface-level overlap and are brittle to paraphrasing, template wrapping, and minor perturbations. Conversely, purely embedding-based retrieval provides greater semantic sensitivity but is computationally prohibitive at trillion-document scale and requires brittle threshold selection. These challenges are exacerbated by the heterogeneous nature of web corpora, characterized by template-heavy pages, viral text fragments, containment hierarchies, and variable-length content.

Framework Overview

SemHash-LLM proposes a unified, hierarchical deduplication architecture integrating semantic projection hashing, attention-weighted MinHash, contrastive boundary learning, and selective LLM-based adjudication. The system conducts multi-granularity feature extraction—encompassing character, token, and semantic levels—combined through a gated fusion mechanism. Deduplication is cast across five key document types: template-wrapped near-duplicates, short text perturbations, hot-spot skew, parent-child containment, and viral fragments. The cascaded pipeline delivers significant computational reductions by reserved neural verification for only ~0.7% of all candidate pairs, without compromising detection quality.

Figure 1: SemHash-LLM pipeline, which fuses multi-granularity features and cascades efficient hashing and decision mechanisms for end-to-end deduplication across diverse redundancy types.

Semantic Projection Hashing

A principal innovation is the Semantic Projection Hashing (SPH) mechanism, which learns binary hash codes optimized in the LLM embedding space. SPH employs a distilled student encoder to generate embeddings amenable to efficient downstream hashing, leveraging mean pooling for robust semantics. The binary hashing functions are parameterized as hyperplanes and trained to maximize similarity preservation while enforcing bit loss balance and mutual orthogonality among hash functions.

Figure 2: SPH module: student embeddings from the LLM teacher are mapped into compact, semantically-optimized binary codes via learned hyperplane partitions.

This approach directly addresses the inability of conventional LSH to represent semantic equivalence; instead, it aligns Hamming neighborhoods with cosine proximity in the semantic space. The training loss combines similarity preservation, orthogonality, and bit balance to ensure high code utilization and minimal false negatives.

Attention-Weighted MinHash

To efficiently suppress spurious matches due to boilerplate or shared templates, the framework introduces Attention-Weighted MinHash (AW-MinHash). This technique aggregates transformer multi-head attention signals to assess within-document token importance, augmenting IDF-weighted n-gram sampling. The weighted MinHash signatures focus representation on document-specific informative content, minimizing the influence of peripheral or duplicated boilerplate text.

Figure 3: AW-MinHash pipeline: transformer attention and IDF weighting emphasize discriminative phrases, with signatures banded adaptively based on attention entropy.

AW-MinHash deploys a novel adaptive LSH banding scheme, tuned dynamically according to the entropy profile of attention distributions. Documents with focused attention profiles utilize fewer bands (higher precision), while those with diffuse attention profiles are filtered with finer granularity. This design enables high recall in the presence of both localized and distributed redundancies.

Contrastive Boundary Learning and LLM-as-Judge

A critical observation is the domain-specificity of optimal duplicate thresholds, which vary with document structure and redundancy type. SemHash-LLM abandons fixed thresholds, instead learning adaptive type-specific boundaries via contrastive learning over fused semantic and lexical features. Monte Carlo dropout quantifies decision uncertainty, supporting precision routing of ambiguous cases to an LLM-as-Judge refinement stream.

Figure 4: Contrastive boundary learning refines pair scoring and delegates high-uncertainty pairs (~3%) to an LLM for final adjudication via structured deliberation.

The LLM-as-Judge is invoked only for borderline cases, leveraging explicit, criteria-driven prompts to render structured duplicate/containment/unique judgments. Final predictions are fused through a confidence-weighted ensemble, where the system adaptively trusts the most certain stream.

Multi-Granularity Fusion and Cascaded Filtering

To maintain recall across diverse duplication modes, SemHash-LLM employs a multi-granularity fusion network. Character-, token-, and semantic-level features are extracted with independent modules—multi-scale CNN, BiLSTM, and MLP over LLM embeddings—then fused with context-sensitive gating that weights each granularity according to signal content and redundancy type.

The candidate reduction strategy is materialized as a cascade: Bloom Filter prescreening, semantic hash blocking, attention-weighted MinHash banding, and final multi-granularity neural verification. Each stage yields a sharp reduction in candidate pairs, ultimately resulting in neural evaluation for less than 1% of all candidates—a notable advance in efficiency.

Figure 5: The cascaded filtering pipeline achieves sublinear scaling in neural compute while realizing high pruning rates at each stage.

Containment and Fragment Detection

SemHash-LLM addresses hierarchical containment and fragment viralization via semantic-augmented n-gram overlap and suffix array indexing, respectively. Containment detection requires both n-gram coverage and semantic confirmation. For viral fragments, a semantic relevance score determines whether a high-frequency substring represents meaningful content or merely boilerplate, guiding retention or removal accordingly.

Experimental Results

SemHash-LLM demonstrates superior empirical results on a 100GB RedPajama web corpus. Across five deduplication categories, the system achieves a composite final score of 91.05, substantially surpassing SimHash, MinHash-LSH, NearDup-BERT, and DedupLM baselines. Ablation studies confirm the necessity of each proposed module (SPH, AW-MinHash, and MGFN) for optimal performance, with removal dropping final scores by 6–12 points.

Strong numerical highlights:

• $S_A$ (template-wrapped): 0.94; $S_C$ (character perturbation): 0.89; $S_D$ (containment): 0.92
• Final neural verification cost kept under 1% of all candidates

Implications and Future Developments

Practically, SemHash-LLM provides a scalable, robust foundation for corpus curation at petabyte scale, directly benefiting LLM pretraining by minimizing data redundancy and downstream memorization artefacts. Theoretical implications highlight the feasibility of tightly coupling deep semantic understanding with efficient binary hashing for large-scale matching problems. The selective, uncertainty-aware invocation of LLM-based reasoning marks a promising direction for hybrid symbolic-neural decision systems.

Future work could extend the framework to real-time or streaming scenarios, incorporate active retrieval augmentation, and explore tighter integration of meta-data signals. The adaptivity of boundary learning and banding further suggests applications in cross-lingual or multi-modal deduplication, as well as dynamic corpus versioning for continual learning settings.

Conclusion

SemHash-LLM delivers a comprehensive, scalable solution to the document deduplication problem by integrating semantic hashing, attention-modulated lexical sketching, contrastive decision boundaries, and judicious LLM adjudication within a multi-stage pipeline. The system achieves state-of-the-art deduplication quality across diverse redundancy types while maintaining tractable compute at trillionscale, and its modularity and adaptivity offer a template for future work in large-scale semantic data management.