Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
153 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Generative modeling of living cells with SO(3)-equivariant implicit neural representations (2304.08960v2)

Published 18 Apr 2023 in cs.CV, cs.LG, eess.IV, and q-bio.QM

Abstract: Data-driven cell tracking and segmentation methods in biomedical imaging require diverse and information-rich training data. In cases where the number of training samples is limited, synthetic computer-generated data sets can be used to improve these methods. This requires the synthesis of cell shapes as well as corresponding microscopy images using generative models. To synthesize realistic living cell shapes, the shape representation used by the generative model should be able to accurately represent fine details and changes in topology, which are common in cells. These requirements are not met by 3D voxel masks, which are restricted in resolution, and polygon meshes, which do not easily model processes like cell growth and mitosis. In this work, we propose to represent living cell shapes as level sets of signed distance functions (SDFs) which are estimated by neural networks. We optimize a fully-connected neural network to provide an implicit representation of the SDF value at any point in a 3D+time domain, conditioned on a learned latent code that is disentangled from the rotation of the cell shape. We demonstrate the effectiveness of this approach on cells that exhibit rapid deformations (Platynereis dumerilii), cells that grow and divide (C. elegans), and cells that have growing and branching filopodial protrusions (A549 human lung carcinoma cells). A quantitative evaluation using shape features and Dice similarity coefficients of real and synthetic cell shapes shows that our model can generate topologically plausible complex cell shapes in 3D+time with high similarity to real living cell shapes. Finally, we show how microscopy images of living cells that correspond to our generated cell shapes can be synthesized using an image-to-image model.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (70)
  1. Implicit neural representations for modeling of abdominal aortic aneurysm progression. arXiv preprint arXiv:2303.01069 .
  2. Microscopy cell segmentation via convolutional LSTM networks, in: 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), IEEE. pp. 1008–1012.
  3. Frame averaging for equivariant shape space learning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 631–641.
  4. Red blood cell image generation for data augmentation using conditional generative adversarial networks, in: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1039–1048.
  5. Generative adversarial networks for augmenting training data of microscopic cell images. Frontiers in Computer Science 1.
  6. Explicitly disentangling image content from translation and rotation with spatial-VAE. Advances in Neural Information Processing Systems 32.
  7. Influence of synthetic label image object properties on gan supported segmentation pipelines, in: Proceedings 29th Workshop Computational Intelligence, pp. 289–305.
  8. The genetics of Caenorhabditis elegans. Genetics 77, 71–94.
  9. Biomedical Image Synthesis and Simulation: Methods and Applications. Academic Press.
  10. Local implicit neural representations for multi-sequence MRI translation, in: 2023 IEEE 20th International Symposium on Biomedical Imaging (ISBI), (in press).
  11. Effects of interpolation on segmentation in cell imaging. Computación y Sistemas 18, 97–109.
  12. Shape analysis and classification: theory and practice. CRC Press, Inc.
  13. Probing cellular processes by long-term live imaging–historic problems and current solutions. Journal of cell science 126, 3805–3815.
  14. Vector neurons: a general framework for SO(3)-equivariant networks, in: IEEE/CVF International Conference on Computer Vision (ICCV).
  15. Characterization of cell shape and deformation in 3D using spherical harmonics, in: 9th International Symposium on Biomedical Imaging (ISBI), IEEE. pp. 848–851.
  16. Segmenting and tracking fluorescent cells in dynamic 3-D microscopy with coupled active surfaces. IEEE Transactions on Image Processing 14, 1396–1410.
  17. DeepSynth: Three-dimensional nuclear segmentation of biological images using neural networks trained with synthetic data. Scientific reports 9, 1–15.
  18. Hyperdiffusion: Generating implicit neural fields with weight-space diffusion. arXiv preprint arXiv:2303.17015 .
  19. A practical method for constructing equivariant multilayer perceptrons for arbitrary matrix groups, in: Proceedings of the 38 th International Conference on Machine Learning.
  20. Three dimensional fluorescence microscopy image synthesis and segmentation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 2221–2229.
  21. PhysiCell: an open source physics-based cell simulator for 3-D multicellular systems. PLoS computational biology 14, e1005991.
  22. CytoGAN: Generative modeling of cell images. bioRxiv .
  23. Implicit geometric regularization for learning shapes, in: Proceedings of the 37th International Conference on Machine Learning, pp. 3789–3799.
  24. Nuclei counting in microscopy images with three dimensional generative adversarial networks, in: Medical Imaging 2019: Image Processing, International Society for Optics and Photonics. SPIE. pp. 753 – 763.
  25. Image-to-image translation with conditional adversarial networks, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1125–1134.
  26. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 .
  27. A model of the spatial tumour heterogeneity in colorectal adenocarcinoma tissue. BMC bioinformatics 17, 255.
  28. Transfer learning in optical microscopy, in: Svoboda, D., Burgos, N., Wolterink, J.M., Zhao, C. (Eds.), Simulation and Synthesis in Medical Imaging, Springer International Publishing, Cham. pp. 77–86.
  29. Challenges and benchmarks in bioimage analysis. Focus on Bio-Image Informatics, ch. 9 , 231–262Springer.
  30. Computational framework for simulating fluorescence microscope images with cell populations. IEEE transactions on medical imaging 26, 1010–1016.
  31. Three-dimensional simulations of the cell growth and cytokinesis using the immersed boundary method. Mathematical Biosciences 271, 118–127.
  32. Marching cubes: A high resolution 3D surface construction algorithm. ACM SIGGRAPH Computer Graphics 21, 163–169.
  33. Embedtrack—simultaneous cell segmentation and tracking through learning offsets and clustering bandwidths. IEEE Access 10, 77147–77157.
  34. Simulation of bright-field microscopy images depicting pap-smear specimen. Cytometry Part A 87, 212–226.
  35. Cell segmentation: 50 years down the road [life sciences]. IEEE signal processing magazine 29, 140–145.
  36. A bird’s-eye view of deep learning in bioimage analysis. Computational and Structural Biotechnology Journal 18, 2312–2325.
  37. A cell-centered approach to developmental biology. Physica A: Statistical Mechanics and its Applications 352, 113–130.
  38. NeRF: Representing scenes as neural radiance fields for view synthesis, in: European conference on computer vision, Springer. pp. 405–421.
  39. Building cell models and simulations from microscope images. Methods 96, 33–39.
  40. Automated analysis of embryonic gene expression with cellular resolution in C. elegans. Nature Methods 5, 703–709.
  41. GANs for biological image synthesis, in: Proceedings of the IEEE International Conference on Computer Vision, pp. 2233–2242.
  42. DeepSDF: Learning continuous signed distance functions for shape representation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 165–174.
  43. PyTorch: An imperative style, high-performance deep learning library, in: Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., Garnett, R. (Eds.), Advances in Neural Information Processing Systems, Curran Associates, Inc. URL: https://proceedings.neurips.cc/paper_files/paper/2019/file/bdbca288fee7f92f2bfa9f7012727740-Paper.pdf.
  44. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434 .
  45. SimuCell: a flexible framework for creating synthetic microscopy images. Nature methods 9, 634.
  46. Accelerating 3D deep learning with PyTorch3D. arXiv preprint arXiv:2007.08501 .
  47. Gaussian mixture models. Encyclopedia of biometrics 741.
  48. NeRP: implicit neural representation learning with prior embedding for sparsely sampled image reconstruction. IEEE Transactions on Neural Networks and Learning Systems .
  49. Implicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems 33, 7462–7473.
  50. FiloGen: a model-based generator of synthetic 3-D time-lapse sequences of single motile cells with growing and branching filopodia. IEEE Transactions on Medical Imaging 37, 2630–2641.
  51. Morpheus: a user-friendly modeling environment for multiscale and multicellular systems biology. Bioinformatics 30, 1331–1332.
  52. Generating semi-synthetic validation benchmarks for embryomics, in: 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI), IEEE. pp. 684–688.
  53. Cellpose: a generalist algorithm for cellular segmentation. Nature methods 18, 100–106.
  54. CoIL: Coordinate-based internal learning for imaging inverse problems. IEEE Transactions on Computational Imaging 7.
  55. Generation of digital phantoms of cell nuclei and simulation of image formation in 3D image cytometry. Cytometry Part A 75, 494–509.
  56. Image-based simulations of tubular network formation, in: 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI), IEEE. pp. 1608–1612.
  57. MitoGen: A framework for generating 3D synthetic time-lapse sequences of cell populations in fluorescence microscopy. IEEE Transactions on Medical Imaging 36, 310–321.
  58. Multi-scale modeling of tissues using CompuCell3D. Methods in cell biology 110, 325–366.
  59. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems 33, 7537–7547.
  60. Quantitative high-speed imaging of entire developing embryos with simultaneous multiview light-sheet microscopy. Nature methods 9, 755–763.
  61. An objective comparison of cell-tracking algorithms. Nature Methods 14, 1141–1152.
  62. Virtual cell imaging: A review on simulation methods employed in image cytometry. Cytometry Part A 89, 1057–1072.
  63. Simulating tissue mechanics with agent-based models: concepts, perspectives and some novel results. Computational particle mechanics 2, 401–444.
  64. High-resolution image synthesis and semantic manipulation with conditional GANs, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8798–8807.
  65. Assessing technician effects when extracting quantities from microscope images. Journal of microbiological methods 53, 97–106.
  66. On generative modeling of cell shape using 3D GANs, in: International Conference on Image Analysis and Processing, Springer. pp. 672–682.
  67. Implicit neural representations for generative modeling of living cell shapes, in: Wang, L., Dou, Q., Fletcher, P.T., Speidel, S., Li, S. (Eds.), Medical Image Computing and Computer Assisted Intervention – MICCAI 2022, Springer Nature Switzerland, Cham. pp. 58–67.
  68. CytoPacq: A web-interface for simulating multi-dimensional cell imaging. Bioinformatics 35, 4531–4533.
  69. Implicit neural representations for deformable image registration, in: International Conference on Medical Imaging with Deep Learning, PMLR. pp. 1349–1359.
  70. Neural fields in visual computing and beyond, in: Computer Graphics Forum, Wiley Online Library. pp. 641–676.
Citations (10)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now