Papers
Topics
Authors
Recent
2000 character limit reached

ChAda-ViT : Channel Adaptive Attention for Joint Representation Learning of Heterogeneous Microscopy Images (2311.15264v2)

Published 26 Nov 2023 in cs.CV and cs.LG

Abstract: Unlike color photography images, which are consistently encoded into RGB channels, biological images encompass various modalities, where the type of microscopy and the meaning of each channel varies with each experiment. Importantly, the number of channels can range from one to a dozen and their correlation is often comparatively much lower than RGB, as each of them brings specific information content. This aspect is largely overlooked by methods designed out of the bioimage field, and current solutions mostly focus on intra-channel spatial attention, often ignoring the relationship between channels, yet crucial in most biological applications. Importantly, the variable channel type and count prevent the projection of several experiments to a unified representation for large scale pre-training. In this study, we propose ChAda-ViT, a novel Channel Adaptive Vision Transformer architecture employing an Inter-Channel Attention mechanism on images with an arbitrary number, order and type of channels. We also introduce IDRCell100k, a bioimage dataset with a rich set of 79 experiments covering 7 microscope modalities, with a multitude of channel types, and counts varying from 1 to 10 per experiment. Our architecture, trained in a self-supervised manner, outperforms existing approaches in several biologically relevant downstream tasks. Additionally, it can be used to bridge the gap for the first time between assays with different microscopes, channel numbers or types by embedding various image and experimental modalities into a unified biological image representation. The latter should facilitate interdisciplinary studies and pave the way for better adoption of deep learning in biological image-based analyses. Code and Data available at https://github.com/nicoboou/chadavit.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (65)
  1. Cnn-based cell analysis: From image to quantitative representation. Frontiers in Bioengineering and Biotechnology, 9:673840, 2021.
  2. Self-supervised learning from images with a joint-embedding predictive architecture. In ICCV, 2023.
  3. Irwan Bello. Lambdanetworks : Modeling long-range interactions without attention. In ICLR, 2021.
  4. No free lunch in self supervised representation learning, 2023.
  5. Unpaired image-to-image translation with limited data to reveal subtle phenotypes. In 2023 IEEE 20th International Symposium on Biomedical Imaging (ISBI), pages 1–5. IEEE, 2023.
  6. Mark-Anthony Bray and Sigrun M et al. Gustafsdottir. A dataset of images and morphological profiles of 30 000 small-molecule treatments using the Cell Painting assay. GigaScience, 6(12):giw014, 2017.
  7. High-Content Phenotypic Profiling of Drug Response Signatures across Distinct Cancer Cells. Molecular Cancer Therapeutics, 9(6):1913–1926, 2010.
  8. End-to-end object detection with transformers. In ECCV, 2020.
  9. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
  10. Jump cell painting dataset: morphological impact of 136,000 chemical and genetic perturbations. bioRxiv, 2023.
  11. Three million images and morphological profiles of cells treated with matched chemical and genetic perturbations. bioRxiv, 2022.
  12. Transunet: Transformers make strong encoders for medical image segmentation. ArXiv, abs/2102.04306, 2021.
  13. Aura-net: Robust segmentation of phase-contrast microscopy images with few annotations. 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pages 640–644, 2021.
  14. Cell painting transfer increases screening hit rate. bioRxiv, 2022.
  15. Label-free prediction of cell painting from brightfield images. Scientific Reports, 12, 2021.
  16. Up-detr: Unsupervised pre-training for object detection with transformers. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1601–1610, 2020.
  17. An image is worth 16x16 words: Transformers for image recognition at scale. ArXiv, abs/2010.11929, 2020.
  18. Unbiased single-cell morphology with self-supervised vision transformers. bioRxiv, 2023a.
  19. Dinov2: Learning robust visual features without supervision, 2023b.
  20. Scaling self-supervised learning for histopathology with masked image modeling. medRxiv, 2023.
  21. Weakly supervised cross-model learning in high-content screening, 2023.
  22. Deep profiling of mouse splenic architecture with codex multiplexed imaging. Cell, 174:968 – 981.e15, 2017.
  23. Leveraging ai transfer learning for rapid and accurate identification and quantification of cellular biomarkers in microscopy images. The FASEB Journal, 36, 2022.
  24. The bioimage archive - building a home for life-sciences microscopy data. bioRxiv, 2021.
  25. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2015.
  26. Identity mappings in deep residual networks. European conference on computer vision, pages 630–645, 2016.
  27. Masked autoencoders are scalable vision learners, 2021.
  28. Alternation of inverse problem approach and deep learning for lens-free microscopy image reconstruction. Scientific Reports, 10(1):20207, 2020.
  29. Spatial mapping of protein composition and tissue organization: a primer for multiplexed antibody-based imaging. Nature Methods, 19:284 – 295, 2021.
  30. Stanley Hua et al. Cytoimagenet: A large-scale pretraining dataset for bioimage transfer learning. In NeuIPS LMRL Workshop, 2021.
  31. Transformers in vision: A survey. ACM Computing Surveys (CSUR), 2021.
  32. Self-supervision advances morphological profiling by unlocking powerful image representations. bioRxiv, 2023.
  33. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60:84 – 90, 2012a.
  34. Deep convolutional networks for large-scale image recognition. Neural Information Processing Systems, 2012b.
  35. Revealing invisible cell phenotypes with conditional generative modeling. Nature Communications, 14(1):6386, 2023.
  36. Backpropagation applied to handwritten zip code recognition. Neural Computation, 1:541–551, 1989.
  37. Deep learning. Nature, 521(7553):436–444, 2015.
  38. Vision permutator: A permutable mlp-like architecture for visual recognition. arXiv preprint arXiv:2106.12368, 2021.
  39. Challenges and opportunities in bioimage analysis. Nature Methods, 20(4):367–377, 2023. https://www.nature.com/articles/s41592-023-01900-4.
  40. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103.14030, 2021.
  41. Comparison of semi-supervised learning methods for high content screening quality control. In ECCV Workshops, 2022.
  42. Cellprofiler 3.0: Next-generation image processing for biology. PLoS Biol, 16(7):e2005970, 2018.
  43. Adavit: Adaptive vision transformers for efficient image recognition. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12299–12308, 2021.
  44. Predicting compound activity from phenotypic profiles and chemical structures. Nature Communications, 14(1):1967, 2023.
  45. Do vision transformers see like convolutional neural networks? https://ar5iv.org/abs/2108.08810, 2021.
  46. Dynamicvit: Efficient vision transformers with dynamic token sparsification. ArXiv, abs/2106.02034, 2021.
  47. Toward causal representation learning. Proceedings of the IEEE, 109:612–634, 2021.
  48. Mitformer: A multiinstance vision transformer for remote sensing scene classification. IEEE Geoscience and Remote Sensing Letters, 19:1–5, 2022.
  49. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
  50. Convnets match vision transformers at scale, 2023.
  51. Going deeper with convolutions. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1–9, 2015.
  52. Distributed computing in practice: the condor experience. Concurrency - Practice and Experience, 17(2-4):323–356, 2005.
  53. Training data-efficient image transformers & distillation through attention. PMLR, 2021a.
  54. Going deeper with image transformers. In ICCV, 2021b.
  55. Attention is all you need. In Neural Information Processing Systems, 2017.
  56. Democratising deep learning for microscopy with zerocostdl4mic. Nature Communications, 12, 2021.
  57. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In ICCV, 2021.
  58. Spatially separable self-attention for vision transformers. arXiv preprint arXiv:2206.03563, 2022.
  59. Eleanor Williams and Josh et al. Moore. Image data resource: a bioimage data integration and publication platform. Nature Methods, 14(8):775–781, 2017.
  60. Segformer: Simple and efficient design for semantic segmentation with transformers. ArXiv, abs/2105.15203, 2021.
  61. Microsnoop: a generalist tool for the unbiased representation of heterogeneous microscopy images. bioRxiv, 2023.
  62. Medmnist v2 - a large-scale lightweight benchmark for 2d and 3d biomedical image classification. Scientific Data, 10(1):41, 2023.
  63. Mil-vt: Multiple instance learning enhanced vision transformer for fundus image classification. In International Conference on Medical Image Computing and Computer-Assisted Intervention, 2021.
  64. Tokens-to-token vit: Training vision transformers from scratch on imagenet. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 538–547, 2021.
  65. Cross-modal graph contrastive learning with cellular images. bioRxiv, pages 2022–06, 2022.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now

Whiteboard

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Tweets

Sign up for free to view the 1 tweet with 0 likes about this paper.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up for Free