Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
149 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

An Error-Bounded Lossy Compression Method with Bit-Adaptive Quantization for Particle Data (2404.02826v2)

Published 3 Apr 2024 in cs.IT, astro-ph.IM, cs.GR, and math.IT

Abstract: This paper presents error-bounded lossy compression tailored for particle datasets from diverse scientific applications in cosmology, fluid dynamics, and fusion energy sciences. As today's high-performance computing capabilities advance, these datasets often reach trillions of points, posing significant visualization, analysis, and storage challenges. While error-bounded lossy compression makes it possible to represent floating-point values with strict pointwise accuracy guarantees, the lack of correlations in particle data's storage ordering often limits the compression ratio. Inspired by quantization-encoding schemes in SZ lossy compressors, we dynamically determine the number of bits to encode particles of the dataset to increase the compression ratio. Specifically, we utilize a k-d tree to partition particles into subregions and generate ``bit boxes'' centered at particles for each subregion to encode their positions. These bit boxes ensure error control while reducing the bit count used for compression. We comprehensively evaluate our method against state-of-the-art compressors on cosmology, fluid dynamics, and fusion plasma datasets.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (64)
  1. Multilevel techniques for compression and reduction of scientific data—the unstructured case. SIAM Journal on Scientific Computing, 42(2):A1402–A1427, 2020.
  2. Nyx: A massively parallel amr code for computational cosmology. The Astrophysical Journal, 765(1):39, 2013.
  3. HOOMD-blue: A Python package for high-performance molecular dynamics and hard particle Monte Carlo simulations. Computational Materials Science, 173:109363, 2020.
  4. P. Angelov. Anomaly detection based on eccentricity analysis. In Proceedings of 2014 IEEE Symposium on Evolving and Autonomous Learning Systems (EALS), pp. 1–8. IEEE, 2014.
  5. J. S. Bagla. Cosmological n-body simulation: Techniques, scope and status. Current science, pp. 1088–1100, 2005.
  6. TTHRESH: Tensor compression for multidimensional visual data. IEEE Transactions on Visualization and Computer Graphics, 26(9):2891–2903, 2019.
  7. Do existing measures of Poincaré plot geometry reflect nonlinear features of heart rate variability? IEEE Transactions on Biomedical Engineering, 48(11):1342–1347, 2001.
  8. 3D point cloud compression: A survey. In Proceedings of The 24th International Conference on 3D Web Technology, pp. 1–9, 2019.
  9. Use cases of lossy compression for floating-point data in scientific data sets. The International Journal of High Performance Computing Applications, 33(6):1201–1220, 2019.
  10. C.-S. Chang and S. Ku. Spontaneous rotation sources in a quiescent tokamak edge plasma. Physics of Plasmas, 15(6), 2008.
  11. Objective comparison of particle tracking methods. Nature Methods, 11(3):281–289, 2014.
  12. O. Devillers and P.-M. Gandoin. Geometric compression for interactive transmission. In Proceedings Visualization 2000., pp. 319–326. IEEE, 2000.
  13. Draco: 3D data compression. https://google.github.io/draco/ [Accessed: 2024-03-20].
  14. HACC: Simulating sky surveys on state-of-the-art supercomputing architectures. New Astronomy, 42:49–65, 2016.
  15. Mira-Titan Universe simulation. https://cosmology.alcf.anl.gov/transfer/miratitan [Accessed: 2024-03-08].
  16. STNet: An end-to-end generative framework for synthesizing spatiotemporal super-resolution volumes. IEEE Transactions on Visualization and Computer Graphics, 28(1):270–280, 2021.
  17. K. Heitmann. Timestep 499 of small outer rim, 2019. doi: 10 . 21227/zg3m-8j73
  18. The Mira-Titan Universe: Precision predictions for dark energy surveys. The Astrophysical Journal, 820(2):108, 2016.
  19. HACC cosmological simulations: First data release. The Astrophysical Journal Supplement Series, 244(1):17, 2019.
  20. Progressive tree-based compression of large-scale particle data. IEEE Transactions on Visualization and Computer Graphics, 2023.
  21. D. A. Huffman. A method for the construction of minimum-redundancy codes. Proceedings of the IRE, 40(9):1098–1101, 1952.
  22. Out-of-core compression and decompression of large n-dimensional scalar fields. Computer Graphics Forum, 22(3):343–348, 2003.
  23. IEEE VIS. Scientific Visualization Contest, 2016. http://sciviscontest.ieeevis.org/2016/ [Accessed: 2024-03-08].
  24. Understanding GPU-based lossy compression for extreme-scale cosmological simulations. In Proceedings of 2020 IEEE International Parallel and Distributed Processing Symposium (IPDPS), pp. 105–115. IEEE, 2020.
  25. A review of advancements in coarse-grained molecular dynamics simulations. Molecular Simulation, 47(10-11):786–803, 2021.
  26. Compressing the incompressible with ISABELA: In-situ reduction of spatio-temporal data. In Proceedings of Euro-Par 2011 Parallel Processing: 17th International Conference, Euro-Par 2011, Bordeaux, France, August 29-September 2, 2011, Proceedings, Part I 17, pp. 366–379. Springer, 2011.
  27. Lossy scientific data compression with SPERR. In Proceedings of 2023 IEEE International Parallel and Distributed Processing Symposium (IPDPS), pp. 1007–1017. IEEE, 2023.
  28. Error-controlled lossy compression optimized for high compression ratios of scientific datasets. In Proceedings of 2018 IEEE International Conference on Big Data (Big Data), pp. 438–447, 2018.
  29. SZ3: A modular framework for composing prediction-based error-bounded lossy compressors. IEEE Transactions on Big Data, 9(2):485–498, 2022.
  30. P. Lindstrom. Fixed-rate compressed floating-point arrays. IEEE Transactions on Visualization and Computer Graphics, 20(12):2674–2683, 2014.
  31. P. Lindstrom and M. Isenburg. Fast and efficient compression of floating-point data. IEEE Transactions on Visualization and Computer Graphics, 12(5):1245–1250, 2006.
  32. L. Lista. Statistical methods for data analysis in particle physics, vol. 909. Springer, 2016.
  33. Exploring autoencoder-based error-bounded compression for scientific data. In Proceedings of 2021 IEEE International Conference on Cluster Computing (CLUSTER), pp. 294–306, 2021.
  34. Compressive neural representations of volumetric scalar fields. Computer Graphics Forum, 40:135–146, 2021.
  35. J. S. Marshall. Modeling and sensitivity analysis of particle impact with a wall with integrated damping mechanisms. Powder Technology, 339:17–24, 2018.
  36. ACORN: Adaptive coordinate networks for neural scene representation. ACM Transactions on Graphics (TOG), 40(4):1–13, 2021.
  37. Methods for cell and particle tracking. Methods in Enzymology, 504:183–200, 2012.
  38. D. Nečas and P. Klapetek. Gwyddion: An open-source software for SPM data analysis. Open Physics, 10(1):181–188, 2012.
  39. Scalable I/O of large-scale molecular dynamics simulations: A data-compression algorithm. Computer Physics Communications, 131(1-2):78–85, 2000.
  40. Direct detection of galactic halo dark matter. Science, 292(5517):698–702, 2001.
  41. Accelerated molecular dynamics methods: Introduction and recent developments. Annual Reports in Computational Chemistry, 5:79–98, 2009.
  42. Survey on deep learning-based point cloud compression. Frontiers in Signal Processing, 2:846972, 2022.
  43. Voxelcontext-net: An octree based framework for point cloud compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6042–6051, 2021.
  44. A prediction-traversal approach for compressing scientific data on unstructured meshes with bounded error. Eurographics Conference on Visualization (EuroVis) [Accepted], 2024.
  45. Particle-based volume rendering. In Proceedings of 2007 6th International Asia-Pacific Symposium on Visualization, pp. 129–132. IEEE, 2007.
  46. Emerging MPEG standards for point cloud compression. IEEE Journal on Emerging and Selected Topics in Circuits and Systems, 9(1):133–148, 2018.
  47. K. Sims. Particle animation and rendering using data parallel computation. In Proceedings of the 17th Annual Conference on Computer Graphics and Interactive Techniques, pp. 405–413, 1990.
  48. Numerical simulation and sensitivity analysis of detailed soot particle size distribution in laminar premixed ethylene flames. Combustion and Flame, 145(1-2):117–127, 2006.
  49. Implicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33:7462–7473, 2020.
  50. Study of a complex fluid-structure dam-breaking benchmark problem using a multi-phase sph method with apr. Engineering Analysis with Boundary Elements, 104:240–258, 2019.
  51. In-depth exploration of single-snapshot lossy compression techniques for N-body simulations. In Proceedings of 2017 IEEE International Conference on Big Data (Big Data), pp. 486–493. IEEE, 2017.
  52. Significantly improving lossy compression for scientific data sets based on multidimensional prediction and error-controlled quantization. In Proceedings of 2017 IEEE International Parallel and Distributed Processing Symposium (IPDPS), pp. 1129–1139. IEEE, 2017.
  53. G. Turk. Interactive collision detection for molecular graphics. Master’s thesis, University of North Carolina at Chapel Hill, 1989.
  54. J. P. Verboncoeur. Particle simulation of plasmas: Review and advances. Plasma Physics and Controlled Fusion, 47(5A):A231, 2005.
  55. Fast neural representations for direct volume rendering. Computer Graphics Forum, 41:196–211, 2022.
  56. H. Wiman and Y. Qin. Fast compression and access of lidar point clouds using wavelets. In Proceedings of 2009 Joint Urban Remote Sensing Event, pp. 1–6. IEEE, 2009.
  57. Deep hierarchical super resolution for scientific data. IEEE Transactions on Visualization and Computer Graphics, 2022.
  58. Neural fields in visual computing and beyond. Computer Graphics Forum, 41:641–676, 2022.
  59. Self-supervised learning for point cloud data: A survey. Expert Systems with Applications, p. 121354, 2023.
  60. A multi-branch decoder network approach to adaptive temporal data selection and reconstruction for big scientific simulation data. IEEE Transactions on Big Data, 8(6):1637–1649, 2021.
  61. Optimizing error-bounded lossy compression for scientific data by dynamic spline interpolation. In Proceedings of 2021 IEEE 37th International Conference on Data Engineering (ICDE), pp. 1643–1654. IEEE, 2021.
  62. Significantly improving lossy compression for HPC datasets with second-order prediction and parameter optimization. In Proceedings of the 29th International Symposium on High-Performance Parallel and Distributed Computing, pp. 89–100, 2020.
  63. MDZ: An efficient error-bounded lossy compressor for molecular dynamics. In Proceedings of 2022 IEEE 38th International Conference on Data Engineering (ICDE), pp. 27–40. IEEE, 2022.
  64. ZSTD. http://www.zstd.net [Accessed: 2023-11-15].

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets