Papers
Topics
Authors
Recent
2000 character limit reached

Pruning neural network models for gene regulatory dynamics using data and domain knowledge (2403.04805v2)

Published 5 Mar 2024 in cs.LG, q-bio.QM, stat.AP, and stat.ML

Abstract: The practical utility of machine learning models in the sciences often hinges on their interpretability. It is common to assess a model's merit for scientific discovery, and thus novel insights, by how well it aligns with already available domain knowledge--a dimension that is currently largely disregarded in the comparison of neural network models. While pruning can simplify deep neural network architectures and excels in identifying sparse models, as we show in the context of gene regulatory network inference, state-of-the-art techniques struggle with biologically meaningful structure learning. To address this issue, we propose DASH, a generalizable framework that guides network pruning by using domain-specific structural information in model fitting and leads to sparser, better interpretable models that are more robust to noise. Using both synthetic data with ground truth information, as well as real-world gene expression data, we show that DASH, using knowledge about gene interaction partners within the putative regulatory network, outperforms general pruning methods by a large margin and yields deeper insights into the biological systems being studied.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (52)
  1. Beyond predictions in neural odes: Identification and interventions. arXiv preprint arXiv:2106.12430, 2021.
  2. Sparsity in continuous-depth neural networks. Advances in Neural Information Processing Systems, 35:901–914, 2022.
  3. On the optimization of deep networks: Implicit acceleration by overparameterization. In International Conference on Machine Learning, pp.  244–253, 2018.
  4. Differential co-expression-based detection of conditional relationships in transcriptional data: comparative analysis and application to breast cancer. Genome biology, 20(1):1–21, 2019.
  5. Burkholz, R. Most activation functions can win the lottery without excessive depth. In Advances in Neural Information Processing Systems, 2022.
  6. Explorability and the origin of network sparsity in living systems. Scientific reports, 7(1):12323, 2017.
  7. Adjoint sensitivity analysis for differential-algebraic equations: The adjoint dae system and its numerical solution. SIAM journal on scientific computing, 24(3):1076–1089, 2003.
  8. Provable benefits of overparameterization in model compression: From double descent to pruning neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp.  6974–6983, 2021.
  9. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
  10. Deepvelo: Single-cell transcriptomic deep velocity field learning with neural ordinary differential equations. Science Advances, 8(48):eabq3745, 2022.
  11. Remap 2018: an updated atlas of regulatory regions from an integrative analysis of dna-binding chip-seq experiments. Nucleic acids research, 46(D1):D267–D275, 2018.
  12. Cybenko, G. Approximation by superpositions of a sigmoidal function. Math. Control. Signals Syst., 2(4):303–314, 1989. doi: 10.1007/BF02551274. URL https://doi.org/10.1007/BF02551274.
  13. Proving the lottery ticket hypothesis for convolutional neural networks. In International Conference on Learning Representations, 2022.
  14. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897):414–419, Feb 2022.
  15. Strong time dependence of the 76-gene prognostic signature for node-negative breast cancer patients in the transbig multicenter independent validation series. Clinical cancer research, 13(11):3207–3214, 2007.
  16. Transcriptomic forecasting with neural ordinary differential equations. Patterns, 4(8), 2023.
  17. A general framework for proving the equivariant strong lottery ticket hypothesis. In International Conference on Learning Representations, 2023.
  18. Plant ’n’ seek: Can you find the winning ticket? In International Conference on Learning Representations, 2022.
  19. Towards strong pruning for lottery tickets with non-zero biases. arXiv preprint arXiv:2110.11150, 2021.
  20. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635, 2018.
  21. Pruning neural networks at initialization: Why are we missing the mark? In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=Ig-VyQc-MLK.
  22. Masks, signs, and learning rate rewinding. In International Conference on Learning Representations, 2024.
  23. Learning causal networks using inducible transcription factors and transcriptome-wide time series. Mol Syst Biol, 16(3):e9174, Mar 2020.
  24. The elements of statistical learning: data mining, inference, and prediction, volume 2. Springer, 2009.
  25. Biologically informed NeuralODEs for genome-wide regulatory dynamics. bioRxiv, pp.  2023–02, 2023.
  26. FOXO transcription factor family in cancer and metastasis. Cancer Metastasis Rev, 39(3):681–709, Sep 2020.
  27. Highly accurate protein structure prediction with AlphaFold. Nature, 596(7873):583–589, Aug 2021.
  28. Soft threshold weight reparameterization for learnable sparsity. In International Conference on Machine Learning, pp.  5544–5555. PMLR, 2020.
  29. Li, Q. sctour: a deep learning architecture for robust inference and accurate prediction of cellular dynamics. Genome Biology, 24(1):1–33, 2023.
  30. The molecular signatures database hallmark gene set collection. Cell systems, 1(6):417–425, 2015.
  31. Sparse flows: Pruning continuous-depth models. Advances in Neural Information Processing Systems, 34:22628–22642, 2021.
  32. Dynamic sparse training: Find efficient sparse network from scratch with trainable masked layers. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=SJlbGJrtDB.
  33. Dynamical systems model of rna velocity improves inference of single-cell trajectory, pseudo-time and gene regulation. Journal of Molecular Biology, 434(15):167606, 2022.
  34. Seeing Is Believing: Brain-Inspired Modular Training for Mechanistic Interpretability. Entropy (Basel), 26(1), Dec 2023.
  35. Learning sparse neural networks through L0subscript𝐿0L_{0}italic_L start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT regularization. arXiv preprint arXiv:1712.01312, 2017.
  36. Learning sparse neural networks through l_0 regularization. In International Conference on Learning Representations, 2018.
  37. A sandbox for prediction and integration of dna, rna, and proteins in single cells. In 35th Conference on Neural Information Processing Systems (NeurIPS 2021) Track on Datasets and Benchmarks, 2021.
  38. Proving the lottery ticket hypothesis: Pruning is all you need. In III, H. D. and Singh, A. (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp.  6682–6691. PMLR, 13–18 Jul 2020. URL https://proceedings.mlr.press/v119/malach20a.html.
  39. Role of p53 in breast cancer progression: An insight into p53 targeted therapy. Theranostics, 13(4):1421–1442, 2023.
  40. Computational modeling of biochemical networks using copasi. Systems Biology, pp.  17–59, 2009.
  41. Logarithmic pruning is all you need. In Advances in Neural Information Processing Systems, 2020.
  42. Unmasking the lottery ticket hypothesis: What’s encoded in a winning ticket’s mask? In International Conference on Learning Representations, 2023.
  43. Optimal lottery tickets via subset sum: Logarithmic over-parameterization is sufficient. In Advances in Neural Information Processing Systems, volume 33, pp.  2599–2610, 2020.
  44. Mapping transcriptomic vector fields of single cells. Cell, 185(4):690–711, 2022.
  45. What’s hidden in a randomly weighted neural network? In Conference on Computer Vision and Pattern Recognition, 2020.
  46. Winning the lottery with continuous sparsification. In Advances in Neural Information Processing Systems, 2020.
  47. Rare gems: Finding lottery tickets at initialization. In Advances in Neural Information Processing Systems, 2022.
  48. Inferring latent temporal progression and regulatory networks from cross-sectional transcriptomic data of cancer samples. PLoS Comput Biol, 17(3):e1008379, Mar 2021.
  49. Learning transcriptional and regulatory dynamics driving cancer cell plasticity using neural ode-based optimal transport. bioRxiv, pp.  2023–03, 2023.
  50. Gene regulatory network inference as relaxed graph matching. AAAI Conference on Artificial Intelligence, 2021.
  51. Generative modeling of single-cell time series with prescient enables prediction of cell trajectories with interventions. Nature communications, 12(1):3222, 2021.
  52. Deconstructing lottery tickets: Zeros, signs, and the supermask. In Advances in Neural Information Processing Systems, pp.  3597–3607, 2019.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Slide Deck Streamline Icon: https://streamlinehq.com

Whiteboard

Dice Question Streamline Icon: https://streamlinehq.com

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
Lightbulb Streamline Icon: https://streamlinehq.com

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets

Sign up for free to view the 3 tweets with 10 likes about this paper.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up for Free