Efficient Discovery of Conditional Dependencies with Desbordante

Abstract: Conditional functional dependencies (CFDs) are functional dependencies with a restricted scope: they specify the context in which a dependency holds and are useful for data-quality tasks, specifying complex integrity constraints, and extracting valuable insights from data. We study the CFD discovery problem, which is computationally demanding. We build on the state-of-the-art CFDFinder algorithm and introduce a set of algorithmic and engineering improvements, including a parallelization strategy, to produce ParCFDFinder. Our implementation is integrated into Desbordante - a high-performance open-source data profiler written in C++ that exposes a Python interface, enabling CFD discovery to be invoked from any Python program. Experimental results show that our enhancements speed up the algorithm by up to $318\times$ ($118\times$ on average) and reduce memory usage by up to $23\times$ ($14\times$ on average) compared with the existing Java-based implementation of Metanome. Integrating ParCFDFinder into Desbordante makes it possible, for the first time, to conveniently discover CFDs on datasets with hundreds of thousands of rows on a commodity machine within a reasonable time.

• The paper presents ParCFDFinder, a parallel CFD discovery framework that achieves up to 318× speedup over Java baselines.
• It utilizes efficient C++ data structures and vectorized batch processing to reduce memory usage by up to 23× while optimizing candidate filtering.
• The approach demonstrates linear scalability with increasing rows and columns, enabling practical CFD mining in large-scale, real-time data quality applications.

Efficient CFD Discovery Using ParCFDFinder in Desbordante

Introduction to Conditional Functional Dependencies and Discovery Challenges

Conditional Functional Dependencies (CFDs) generalize classical Functional Dependencies (FDs) by adding context-specific predicates, enabling nuanced integrity constraints and data quality rules within subpopulations of a dataset. Their practical relevance spans data cleaning, outlier and typo detection, schema matching, and feature engineering. However, the discovery problem for CFDs is both algorithmically and computationally challenging, especially as datasets grow in cardinality (rows) and arity (columns).

Previous baseline frameworks, notably Metanome, include a Java implementation (CFDFinder) that, while accurate, suffers in terms of scalability, execution time, and memory footprint. Reimplementations in lower-level languages have shown moderate improvements for earlier dependency discovery primitives, but effective algorithmic optimizations and parallelization for CFD mining had remained largely unexplored.

The ParCFDFinder Algorithmic Architecture and Innovations

The paper presents ParCFDFinder, a high-performance, parallel CFD discovery framework, tightly integrated into the C++ profiler Desbordante with a native Python interface. ParCFDFinder builds atop CFDFinder—preserving its top-down lattice traversal, candidate filtering, and heuristic tableau construction—but incorporates critical engineering and algorithmic enhancements:

• Efficient Data Structures: Substitution of handle-heavy Java containers (PriorityQueue, HashSet) with their high-performance C++ analogues (boost::multi_index, boost::unordered_flat_set) achieves improved cache locality and constant-time lookups.
• Optimized Pattern Generation: Redundant child pattern generation is eliminated by deduplication of candidate constants per attribute within a parent pattern's cluster cover. Cover computation leverages bitmask-based pruning, deferring materialization until post-validation, thus reducing unnecessary memory allocations.
• Batch Processing Pipelines: Batchwise candidate filtering and mask computations exploit vectorized access patterns, minimizing cache misses and facilitating compiler optimizations.
• Parallel Lattice Traversal: Tableau generation for CFD candidates at the same lattice level is parallelized using a thread pool (Boost.Asio), achieving task-level concurrency without introducing per-thread memory bottlenecks.

Figure 1: Schematic representation of the CFDFinder algorithm underlying ParCFDFinder, illustrating preprocessing, traversal, and pattern generation stages.

These enhancements directly address the principal bottleneck—pattern tableau generation, which accounts for the overwhelming majority of runtime in baseline implementations.

Quantitative Evaluation: Runtime, Memory, and Scalability

The empirical evaluation benchmarks ParCFDFinder (single-threaded and parallel variants) against the Metanome (Java) baseline on real and synthetic datasets ranging from hundreds to over 150,000 rows and up to 21 columns. The experiments elucidate three primary axes: execution time, memory efficiency, and parallel scalability.

Numerical outcomes include:

• Speedup: Up to $318\times$ acceleration (average $118\times$) over the Java baseline; speedup is most pronounced for large-row-count datasets that push the Java implementation beyond practical resource or timeout limits.
• Memory reduction: Up to $23\times$ (average $14\times$) lower RAM consumption; C++-specific optimizations and elimination of JVM overhead markedly decrease memory usage, particularly for datasets with high candidate cardinality.
• Parallel scaling: Near-ideal logarithmic speedup as thread count increases, with multi-threaded C++ implementations outperforming both Java and single-threaded C++ consistently across all benchmarks. Importantly, the parallelization does not induce significant memory overhead, as dominant allocations reside in a global support map, not thread-local structures.

Figure 2: Row scalability up to 20,000 rows—C++ implementations sustain linear scaling, while Java terminates early due to timeouts.

Figure 3: Full dataset row scalability (up to 150,935 rows)—C++ (including parallel ParCFDFinder) achieves efficient linear scaling even on large instances.

Figure 4: Column scalability—Exponential runtime growth is intrinsic, but C++ implementations can process up to 19 columns (Java times out at 15).

Figure 5: Thread scalability—Speedup vs. core count shows near-logarithmic increase, with memory usage almost flat as thread count grows.

Theoretical and Practical Implications

The introduction of efficient, scalable CFD mining in commodity environments broadens the range of deployable data quality techniques in production and research contexts. Practically, ParCFDFinder enables real-time profiling and error detection for datasets at scales that were previously infeasible for CFD enumeration, notably supporting pipelines for machine learning data curation, ETL validation, and scientific data integrity assessment.

Theoretically, the batch-oriented candidate filtering and deferred cover materialization suggest a generalizable paradigm for combinatorial constraint mining tasks that share heavy candidate expansion and support evaluation stages. The measured linearity in row scaling also confirms the viability of attribute-domain partitioning and support-based pruning in high-volume settings. The success of parallel lattice traversal and the flat memory profile indicate that further increases in available computational cores could translate directly to throughput gains with minimal coordination or synchronization penalty.

Future Directions

Potential enhancements include integrating additional CFD discovery strategies (e.g., approximate or noisy CFD mining, multi-modal attribute expansions), leveraging non-volatile memory for candidate caching in extremely large datasets, and algorithmic hybridization with pattern mining techniques. The exported Python interface suggests opportunities for seamless integration with popular data science toolchains, expanding the accessibility of dependency-driven feature engineering and dataset auditing. Comparative analyses with data profilers outside the Metanome/Desbordante family may further contextualize the limits of current optimizations.

Conclusion

ParCFDFinder, as implemented in Desbordante, brings state-of-the-art efficiency and scalability to CFD discovery on relational data. Through algorithmic innovations and modern systems-level engineering, it achieves up to $318\times$ runtime and $23\times$ memory improvements relative to prior art, with robust parallel scaling and end-user accessibility from Python. The contributions promise not only to extend CFD usage in downstream applications but also to inspire analogous improvements across pattern-mining and data-quality analytics domains.