High-Performance Sorting-Based k-mer Counting in Distributed Memory with Flexible Hybrid Parallelism (2407.07718v1)

Abstract: In generating large quantities of DNA data, high-throughput sequencing technologies require advanced bioinformatics infrastructures for efficient data analysis. k-mer counting, the process of quantifying the frequency of fixed-length k DNA subsequences, is a fundamental step in various bioinformatics pipelines, including genome assembly and protein prediction. Due to the growing volume of data, the scaling of the counting process is critical. In the literature, distributed memory software uses hash tables, which exhibit poor cache friendliness and consume excessive memory. They often also lack support for flexible parallelism, which makes integration into existing bioinformatics pipelines difficult. In this work, we propose HySortK, a highly efficient sorting-based distributed memory k-mer counter. HySortK reduces the communication volume through a carefully designed communication scheme and domain-specific optimization strategies. Furthermore, we introduce an abstract task layer for flexible hybrid parallelism to address load imbalances in different scenarios. HySortK achieves a 2-10x speedup compared to the GPU baseline on 4 and 8 nodes. Compared to state-of-the-art CPU software, HySortK achieves up to 2x speedup while reducing peak memory usage by 30% on 16 nodes. Finally, we integrated HySortK into an existing genome assembly pipeline and achieved up to 1.8x speedup, proving its flexibility and practicality in real-world scenarios.

• The paper introduces a novel sorting-based approach that optimizes memory usage and speed for k-mer counting in distributed memory systems.
• The paper implements an enhanced supermer strategy with optimized minimizer selection to lower communication overhead in large-scale genomic analyses.
• The paper demonstrates flexible hybrid parallelism using MPI and OpenMP, achieving speedups of 2-10x and up to 30% reduced memory usage compared to existing solutions.

High-Performance Sorting-Based $k$-mer Counting in Distributed Memory with Flexible Hybrid Parallelism

The paper "High-Performance Sorting-Based $k$-mer Counting in Distributed Memory with Flexible Hybrid Parallelism" by Yifan Li and Giulia Guidi presents HySortK, an advanced tool for efficient $k$-mer counting in distributed memory systems tailored for large-scale genomic datasets. This work stands out due to its robust methodological innovations and practical performance improvements, enhancing the efficacy of $k$-mer counting, which is fundamental to numerous bioinformatics applications.

Key Contributions

1. Innovative Sorting-Based Approach: HySortK introduces a novel radix sort-based methodology for $k$-mer counting in distributed systems. This deviates from traditional hash table-based methods, which tend to suffer from poor cache utilization and high memory demands. The sorting-based approach significantly reduces memory usage and improves overall performance.
2. Enhanced Supermer Strategy: The utilization of the supermer technique, an approach to group $k$-mers with common features, minimizes communication overhead. The authors enhance this by employing an optimized method for determining minimizers, which are key in supermer partitioning, further balancing computational load and reducing communication volume.
3. Hybrid Parallelism with Task Abstraction Layer: By incorporating a flexible task abstraction layer that supports both MPI and OpenMP parallelism, HySortK efficiently addresses load imbalances and scales effectively across numerous cores. This hybrid approach ensures that computational resources are optimally utilized, even in complex NUMA architectures.
4. Communication Optimization: The tool implements overlapping of computation and communication and applies domain-specific compression techniques to further trim down communication costs during $k$-mer transactions between nodes.

Empirical Performance and Comparisons

The empirical analysis in the paper highlights the strong numerical performance of HySortK:

• Speedup: HySortK achieves a 2-10x speedup compared to GPU-based alternatives and outperforms state-of-the-art CPU software by up to 2x on several datasets.
• Memory Efficiency: The tool demonstrates peak memory usage reductions by up to 30% when compared to existing solutions.
• Scaling: Both strong and weak scaling results exhibit substantial improvements. For instance, the tool achieves near-perfect scaling efficiency up to a certain threshold of nodes. Moreover, HySortK handles large datasets efficiently, showing significant performance gains in multiple-node configurations.

Practical and Theoretical Implications

The introduction of HySortK marks a noteworthy advancement in the computational biology domain, specifically in the context of $k$-mer counting for genome assembly and other bioinformatics pipelines. Its high performance and low memory footprint make it particularly suitable for large-scale genomic data, which is increasingly prevalent due to advancements in sequencing technologies.

Integration and Future Directions

The successful integration of HySortK into the ELBA genome assembly pipeline underscores its practical applicability. This integration not only ensures faster $k$-mer counting but also leverages the tool's hybrid parallelism to boost the overall pipeline performance.

Future Work:

• Supermer Strategy Enhancement: Future work might focus on further optimizing the supermer strategy, particularly in handling dense genomic regions with heavy repetitions.
• Broader Applications: Extending the methodologies to other bioinformatics tasks and computational domains could also be beneficial.
• Algorithmic Refinements: Continuous refinements in the algorithm, particularly in data compression and load balancing strategies, could yield further performance enhancements.

Conclusion

Overall, HySortK presents a significant step forward in the efficient and scalable $k$-mer counting necessary for modern genomic analysis. It provides a sophisticated combination of algorithmic innovations and practical performance enhancements, bolstered by a thorough empirical evaluation, making it a valuable tool for the computational biology community.