Scalable Maximal Frequent Episode Mining with Desbordante

Abstract: Episode mining aims to extract subsequences of events that possess certain distinctive properties and constitute facts valuable to the user. Maximal frequent episode mining concentrates on discovery of frequently-appearing subsequences, which are not included into any other larger frequent subsequence. The state-of-the-art for this problem is the MaxFEM algorithm which enumerates possible subsequences, while applying various pruning techniques to accelerate the search. However, this is a computationally-intensive problem: reducing the minimum number of required subsequence occurrences or increasing the length of the subsequence both substantially raise running time, which limits practical use of MaxFEM. In this paper we describe our efforts in designing a high-performing algorithm for this problem. For this we: 1) develop an efficient C++ implementation of MaxFEM, and 2) devise an efficient technique to parallelizing it. As the result, we propose an improved parallel MaxFEM variant, which we call ParMaxFEM. Additionally, we integrate the improved algorithm into Desbordante - a high-performance, open-source data profiler with deep Python integration that treats patterns as first-class entities and allows users to develop their custom programs that can include discovery and validation of patterns. To evaluate our approach we compare both C++ implementations with the original SPMF implementation. Experiments demonstrated that our reimplemented version provides up to $8\times$ speedup over the SPMF baseline, while our parallelization technique provides up to $35\times$ improvement overall (on 8 cores).

• The paper presents Desbordante ParMaxFEM, a C++ implementation that enhances maximal frequent episode mining using aggressive algorithmic optimizations and parallelization.
• It demonstrates substantial improvements with up to 35× speedup and 14× lower memory usage compared to traditional Java-based methods.
• The system offers native Python integration for interactive data profiling, making complex episode mining more accessible to researchers.

Scalable Maximal Frequent Episode Mining with Desbordante: A Technical Analysis

Introduction

This paper presents Desbordante ParMaxFEM, a parallel, scalable, C++-based system for maximal frequent episode mining (MaxFEM) in complex event sequences. The authors address weaknesses in the canonical Java-based MaxFEM algorithm—especially the lack of efficiency, parallelism, and seamless Python integration—with new algorithmic engineering, parallelization strategies, and API design. Desbordante integrates the improved algorithm as a core primitive in a high-performance data profiler, showcasing substantial empirical gains over prior methods.

Problem Context and MaxFEM Algorithmic Foundations

Maximal Frequent Episode Mining targets the discovery of all subsequences (episodes) in a single, temporally ordered event sequence that are both frequent (support exceeding a user threshold) and maximal (not strictly included in any other frequent episode). Traditional episode mining approaches over-produce redundant, nested patterns, creating either usability bottlenecks or computational inefficiencies.

The MaxFEM algorithm improves over prior work (e.g., EMMA [10.1016/j.is.2007.07.003]) by using a monotonic DFS strategy, aggressive pruning (Efficient Filtering of Non-maximal Episodes, Skip Extension Checking, and Temporal Pruning), and manipulation of location/bound lists to accelerate the identification of frequent composite episodes. However, the existing Java implementation (in SPMF) is single-threaded, subject to JVM memory management artifacts, and difficult to use within standard data science workflows.

C++ Implementation and System-Level Optimizations

Desbordante's first contribution is a C++ MaxFEM implementation, designed for low-latency, high-throughput mining of complex sequences. Several design strategies are key:

• Cache-Optimized Data Structures: Events are mapped to contiguous integer ranges, allowing minimal-overhead vector-based storage. Preallocation and move semantics minimize allocation and copying overheads.
• Shared Location Lists: Through std::shared_ptr, structurally similar patterns avoid duplicative memory usage—a crucial improvement under the combinatorial explosion typical for lower minimum support levels.
• Explicit Memory Management: The careful manual management of memory (as available in C++) obviates the need for JVM garbage collection, resulting in consistent, low-RSS operation during large workload runs.
• Elimination of Redundant Steps: Recognizing that the re-encoded sequence is no longer required post-location list calculation, Desbordante skips explicit re-encoding, trimming redundant computation.

The resulting implementation accelerates all stages of the pipeline except serial extension (step 5), which is still bottlenecked by explosive search space growth for certain parameter regimes.

Parallelization via ParMaxFEM

ParMaxFEM extends the sequential C++ algorithm to exploit multi-core CPUs. The parallelization is based on the observation that each parallel episode defines an independent compositional subtree, lending itself to coarse-grained parallel DFS. Key aspects include:

• Dynamic Task-Based Thread Pool: The system assigns DFS expansion tasks to a Boost.Asio thread pool, using adaptive splitting to throttle the job queue and control task granularity relative to thread count.
• Thread-Local Result Buffers: Each worker thread collects its local maxima, avoiding lock contention on the global result set; aggregation is deferred until thread completion.
• Batch Filtering and Pruning: Filtering non-maximal episodes is batch-processed post hoc (with hash-based pre-pruning and length-major sorting), reducing synchronization but causing a controlled temporary increase in peak memory usage.

Empirical Impact

Substantial speedup is realized for heavily branching search spaces. The observed gains, including up to $35\times$ acceleration and $14\times$ lower memory consumption (compared to the Java SPMF baseline) on typical datasets, are especially pronounced for larger window lengths and lower support (2607.03188).

Figure 2: Comparative runtime, memory use, and maximal episode counts for multiple MaxFEM implementations on Kosarak and Retail for varying support and window lengths.

Python Integration and Practicality

A significant usability innovation is native Python integration. Desbordante's API allows direct ingestion of Python iterables of event sets (or event sets with timestamps), bypassing brittle text serialization and external JVM process management. This enables in-memory, interactive use (e.g., Jupyter), streaming/generator-based data population, and seamless composition with other numerical tools.

The contrast with SPMF's reliance on text-based CLI invocation and Python subprocess management is stark: the latter approach suffers from high engineering overhead, I/O bottlenecks, and lack of interactivity, severely limiting its adoption by the data science community.

Experimental Evaluation

The authors conduct a comprehensive experimental study over 12 diverse real-world datasets. Benchmarks are performed under tightly controlled hardware isolation to ensure validity of multi-core scalability measurements.

Runtime and Memory Efficiency

C++ Desbordante MaxFEM achieves $2.25\times$ to $2.68\times$ geometric mean speedup over SPMF MaxFEM, with $13.6\times$ to $11.4\times$ lower average memory footprint. ParMaxFEM (8 threads) achieves $5.89\times$ to $11.86\times$ speedup, and up to $34.66\times$ maximum on complex datasets or lower minsup, while remaining under (or at worst, slightly above) the SPMF memory baseline.

Figure 4: Maximum observed speedup for Desbordante MaxFEM and ParMaxFEM across all datasets and parameters.

The speedup scales with thread count, asymptoting at $8.8\times$ on 32 cores. Memory use rises nearly linearly due to thread-local storage, but typically remains below Java/ SPMF baseline, except on certain pathological combinations of dataset and parameter settings.

Figure 1: Parallel scalability of ParMaxFEM: runtime and memory consumption as function of thread count.

Parameter and Dataset Sensitivity

Speedup is highly sensitive to both dataset density and window/minsup parameters. Dense search spaces (e.g., mushrooms, chess) or low minsup yield orders-of-magnitude improvement, while sparser datasets or high support shrinks the relative gains. Memory savings are highest on challenging, memory-bound mining regimes.

Implications and Future Directions

The technical contributions of Desbordante ParMaxFEM are twofold: (1) substantially improved practical tractability of maximal frequent episode mining via low-level engineering and parallelization, and (2) making high-performance mining accessible for the Python-centric analytics ecosystem. In practice, this enables exploration of lower support settings or larger window sizes by default, increasing the applicability of episode mining to real data profiling scenarios such as data cleaning, anomaly/outlier detection, and rule learning.

Theoretically, the parallelization approach follows classic search partitioning—work stealing and thread-local buffering amortize synchronization without loss of maximality, but also introduce limits to perfect scaling per Amdahl’s bound, highlighting that step 5 dominates total mining time, but not all steps are parallelizable.

Further development may include support for incremental, stream-based mining (given Python iterable integration and C++ generator interop), extension to other episode constraints (gap constraints, utility thresholds), and integration with larger-scale distributed frameworks (e.g., Spark, for big data event logs).

Conclusion

Desbordante's C++ ParMaxFEM, as demonstrated in this paper, marks a significant improvement in the tractability and usability of maximal frequent episode mining in single, possibly complex event sequences. Its combination of aggressive algorithmic optimizations, efficient parallelization, and native Python API support make it a reference architecture for future work in episodic sequence pattern mining. These contributions make maximal episode mining feasible for parameter regimes and data sizes previously inaccessible, expanding both the practical and research horizons of pattern discovery in sequential data.