SoftMatcha 2: A Fast and Soft Pattern Matcher for Trillion-Scale Corpora

Abstract: We present an ultra-fast and flexible search algorithm that enables search over trillion-scale natural language corpora in under 0.3 seconds while handling semantic variations (substitution, insertion, and deletion). Our approach employs string matching based on suffix arrays that scales well with corpus size. To mitigate the combinatorial explosion induced by the semantic relaxation of queries, our method is built on two key algorithmic ideas: fast exact lookup enabled by a disk-aware design, and dynamic corpus-aware pruning. We theoretically show that the proposed method suppresses exponential growth in the search space with respect to query length by leveraging statistical properties of natural language. In experiments on FineWeb-Edu (Lozhkov et al., 2024) (1.4T tokens), we show that our method achieves significantly lower search latency than existing methods: infini-gram (Liu et al., 2024), infini-gram mini (Xu et al., 2025), and SoftMatcha (Deguchi et al., 2025). As a practical application, we demonstrate that our method identifies benchmark contamination in training corpora, unidentified by existing approaches. We also provide an online demo of fast, soft search across corpora in seven languages.

• The paper introduces SoftMatcha 2, advancing corpus search with soft pattern matching that accommodates semantic variations, insertions, and deletions.
• It integrates a disk-aware staged suffix array with run-length compression to minimize disk I/O, reducing search latency by up to 33x compared to traditional methods.
• Empirical results demonstrate sub-second latency across multiple languages and effective contamination detection, offering practical benefits for data curation and bias auditing.

Fast and Soft Pattern Matching for Trillion-Scale Corpora: An Expert Analysis of "SoftMatcha 2" (2602.10908)

Overview and Motivation

"SoftMatcha 2: A Fast and Soft Pattern Matcher for Trillion-Scale Corpora" addresses the critical need for efficient, semantically-flexible search algorithms capable of operating on contemporary large-scale corpora—datasets increasingly used to train LLMs and other NLP systems. Traditional methods such as exact matching (suffix arrays, FM-index), regular expressions, and inverted indexing face significant scalability bottlenecks and lack robust mechanisms for handling semantic variations, insertions, and deletions. The SoftMatcha 2 algorithm advances corpus search by integrating disk-aware staged suffix arrays and dynamic corpus-aware pruning, supporting soft pattern matching for trillions of tokens with sub-second latency.

Figure 1: An example of search in SoftMatcha 2, demonstrating soft search over trillion-scale corpora, efficiently capturing word substitution, insertion, and deletion.

Methodological Contributions

Problem Definition and Semantic Relaxation

The algorithm reformulates corpus search as finding token sequences (patterns) with high semantic similarity to queries, accommodating not only substitutions but also insertions and deletions. The similarity metric is constructed from smoothed minima of cosine similarities between word embeddings, further extended to penalize insertions/deletions based on information-theoretic properties (Zipfian whitening). Adaptive thresholding guarantees the output of $K$ matches with meaningful semantic correspondence.

Figure 2: The sketch of the soft searching algorithm, illustrating the search space expansion and subsequent pruning for queries like "olympics gold medal".

Disk-Aware Staged Suffix Arrays

The proposed method overcomes disk I/O bottlenecks via a two-level suffix array: a sparse RAM-resident array guides the search to localized regions in the full index on disk. This design constrains each exact lookup to a single random disk access, which is critical given practical constraints on index size (e.g., >21 TB for FineWeb-Edu, 1.4T tokens). Run-length compression further reduces index footprint, achieving a 7x reduction for typical datasets.

Figure 3: The sketch of the fast suffix array mechanism, illustrating efficient binary search localization for queries.

Dynamic Corpus-Aware Pruning

The exponential candidate explosion induced by semantic relaxation is mitigated through iterative, data-driven pruning. This exploits the power-law distribution of $n$-grams in natural language corpora; each expansion step is filtered for actual corpus occurrences via efficient suffix array lookup. Additional optimizations include $k$-gram existence caching (for frequent short patterns) and "last-bits" pruning (for low-frequency terminal candidates).

Theoretical Analysis

The authors provide formal bounds on lookup operations under statistical assumptions (Zipfian distributions, corpus similarity ratios), demonstrating suppression of exponential growth in search space with respect to query length $m$. For fixed semantic threshold and moderate $m$, the average number of exact lookups does not scale exponentially; further, per Zipf’s law, lookup operations grow sublinearly ($O(\lvert C\rvert^{1/\gamma})$) in corpus size. This explains observed scalability in practice, where latency remains nearly constant across orders-of-magnitude increases in dataset size.

Empirical Results

Latency and Scalability

Experimental evaluation on diverse corpora (up to 1.4T tokens, seven languages) reveals 95th-percentile soft-search latencies of $\le$0.3s across English, Japanese, and Chinese datasets, outperforming prior baselines including SoftMatcha, infini-gram, and infini-gram mini. Exact search latency is reduced by up to 33x compared to infini-gram.

Figure 4: The p95 latency for exact search on FineWeb-Edu, showing superior performance of SoftMatcha 2 versus infini-gram and infini-gram mini.

Figure 5: The p95 latency for soft-search across English, Japanese, and Chinese corpora, with SoftMatcha 2 demonstrating stable and efficient scaling.

Ablation studies confirm the critical role of dynamic pruning; disabling all pruning causes a $>$2x increase in per-token candidate expansion.

Figure 6: Number of exact match lookups with/without pruning over FineWeb-Edu, illustrating exponential suppression from introduced techniques.

Qualitative Evaluation and Multilingual Applicability

Qualitative results across English, Japanese, Chinese, French, German, Italian, and Russian show robust retrieval of semantically-similar sequences, handling synonymy, paraphrase, and variable-length matches missed by existing tools. Case studies present both direct and nuanced matches (e.g., "olympics gold medalist" yielding "olympics silver medalist", variant spellings, and value-substituted templates).

Figure 7: Screenshot of search for "olympics gold medalist", returning 20 high-similarity patterns in 55 ms.

Figure 8: Screenshot of search for "ai may replace fifty percent of jobs", demonstrating extended mode capabilities.

Contamination Detection Application

SoftMatcha 2 is applied to benchmark contamination detection, flagging "dirty" test samples in benchmark datasets overlooked by exact match. Of newly identified samples, 81% are validated as true contamination—semantic paraphrase or template leakage (e.g., number substitution)—highlighting the necessity of soft matching in evaluation data curation.

Figure 9: Number of dirty benchmark problems flagged, breakdown showing new detections uniquely enabled by soft search.

Practical and Theoretical Implications

The SoftMatcha 2 algorithm sets a new standard for semantic corpus search at trillion-scale, with direct implications for training-data analysis, contamination detection, bias auditing, and provenance tracing in contemporary LLM pipelines. Its ability to operate efficiently under high semantic relaxation provides a toolset for investigating unintended model behaviors and supporting data-centric AI methodologies—where understanding training data is crucial for system reliability and fairness.

Theoretically, the approach suggests a tight coupling between statistical linguistic properties (Zipfian regularities) and algorithmic tractability. Future directions include expanding to compositional similarity (multi-token paraphrases), integrating wildcard/wildcard matching, and further reducing resource demands for democratized deployment.

Limitations and Future Directions

Core limitations are:

• Pattern similarity is restricted to word-level embedding, insufficient for mapping compositional paraphrases like "U.S." vs. "United States".
• Algorithmic guarantees presuppose statistical regularity; anomalous corpora or queries may violate observed scaling.
• Resource requirements (RAM/disk) remain non-trivial for trillion-scale corpora.

Planned extensions include incorporation of compositional embedding models, generalized query support (wildcards), and multilingual expansion of the online demo interface. Empirical validation of theoretical claims across wider datasets and linguistic universals is suggested.

Conclusion

"SoftMatcha 2" delivers a scalable, semantically-flexible full-text search engine for trillion-scale corpora, combining disk-aware suffix arrays with dynamic power-law pruning. Strong empirical and theoretical results establish its superiority over prior methods and its utility for practical tasks like contamination detection in benchmark datasets. The approach solidifies the foundation for future advances in efficient, linguistically-informed search algorithms, supporting the evolving demands of AI research and deployment.