Papers
Topics
Authors
Recent
Assistant
AI Research Assistant
Well-researched responses based on relevant abstracts and paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses.
Gemini 2.5 Flash
Gemini 2.5 Flash 63 tok/s
Gemini 2.5 Pro 50 tok/s Pro
GPT-5 Medium 19 tok/s Pro
GPT-5 High 29 tok/s Pro
GPT-4o 101 tok/s Pro
Kimi K2 212 tok/s Pro
GPT OSS 120B 438 tok/s Pro
Claude Sonnet 4.5 36 tok/s Pro
2000 character limit reached

Self-Supervised Learning with Generative Adversarial Networks for Electron Microscopy (2402.18286v2)

Published 28 Feb 2024 in cs.CV, cond-mat.mtrl-sci, cs.AI, and cs.LG

Abstract: In this work, we explore the potential of self-supervised learning with Generative Adversarial Networks (GANs) for electron microscopy datasets. We show how self-supervised pretraining facilitates efficient fine-tuning for a spectrum of downstream tasks, including semantic segmentation, denoising, noise & background removal, and super-resolution. Experimentation with varying model complexities and receptive field sizes reveals the remarkable phenomenon that fine-tuned models of lower complexity consistently outperform more complex models with random weight initialization. We demonstrate the versatility of self-supervised pretraining across various downstream tasks in the context of electron microscopy, allowing faster convergence and better performance. We conclude that self-supervised pretraining serves as a powerful catalyst, being especially advantageous when limited annotated data are available and efficient scaling of computational cost is important.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (60)
  1. Decoding crystallography from high-resolution electron imaging and diffraction datasets with deep learning. Science Advances, 5, 2019.
  2. Self-supervised learning for remote sensing scene classification under the few shot scenario. Scientific Reports, 13, 2023.
  3. Deep learning for computational biology. Molecular Systems Biology, 12, 2016.
  4. Segmentation in large-scale cellular electron microscopy with deep learning: A literature survey. Medical image analysis, pp.  102920, 2023.
  5. Generative pretraining from pixels. In International conference on machine learning, pp. 1691–1703. PMLR, 2020a.
  6. Self-supervised GANs via auxiliary rotation loss. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  12154–12163, 2019.
  7. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597–1607. PMLR, 2020b.
  8. Improved baselines with momentum contrastive learning. ArXiv, abs/2003.04297, 2020c.
  9. Cem500k, a large-scale heterogeneous unlabeled cellular electron microscopy image dataset for deep learning. eLife, 10, April 2021.
  10. Learning speaker embedding with momentum contrast. ArXiv, abs/2001.01986, 2020.
  11. Machine learning for revealing spatial dependence among nanoparticles: understanding catalyst film dewetting via gibbs point process models. The Journal of Physical Chemistry C, 124:27479–27494, 2020.
  12. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
  13. Deep learning of crystalline defects from TEM images: a solution for the problem of ‘never enough training data’. Machine Learning: Science and Technology, 5(1):015006, 2024.
  14. Machine learning pipeline for segmentation and defect identification from high-resolution transmission electron microscopy data. Microscopy and Microanalysis, 27(3):549–556, May 2021.
  15. Self-supervised GANs with similarity loss for remote sensing image scene classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 14:2508–2521, 2021.
  16. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  9729–9738, 2020.
  17. Understanding important features of deep learning models for segmentation of high-resolution transmission electron microscopy images. npj Computational Materials, 6(1), July 2020.
  18. Image-to-Image translation with conditional adversarial networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, July 2017.
  19. Classification of scanning electron microscope images of pharmaceutical excipients using deep convolutional neural networks with transfer learning. International Journal of Pharmaceutics: X, 4:100135, December 2022.
  20. Self-supervised learning for semantic segmentation of archaeological monuments in DTMs. Journal of computer applications in archaeology 6 (2023), Nr. 1, 6(1):155–173, 2023.
  21. Ke, C. Applications of scanning electron microscopy in biology. International Review of Cytology, pp.  183–255, 1971.
  22. EM-net: Deep learning for electron microscopy image segmentation. In 2020 25th International Conference on Pattern Recognition (ICPR). IEEE, January 2021.
  23. Segment anything. arXiv preprint arXiv:2304.02643, 2023.
  24. Statistical characterization of the morphologies of nanoparticles through machine learning based electron microscopy image analysis. ACS Nano, 14:17125–17133, 2020.
  25. Leng, Y. Materials characterization: introduction to microscopic and spectroscopic methods. John Wiley & Sons, 2013.
  26. RGMIM: Region-guided masked image modeling for COVID-19 detection. arXiv e-prints, pp.  arXiv–2211, 2022.
  27. TEMImageNet training library and atomsegnet deep-learning models for high-precision atom segmentation, localization, denoising, and deblurring of atomic-resolution images. Scientific Reports, 11(1), March 2021.
  28. Self-supervised learning is more robust to dataset imbalance. In International Conference on Learning Representations, 2021.
  29. Versatile growth of freestanding orthorhombic α𝛼\alphaitalic_α-molybdenum trioxide nano- and microstructures by rapid thermal processing for gas nanosensors. The Journal of Physical Chemistry C, 118:15068–15078, 2014.
  30. Least squares generative adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp.  2794–2802, 2017.
  31. Deep learning earth observation classification using imagenet pretrained networks. IEEE Geoscience and Remote Sensing Letters, 13(1):105–109, 2015.
  32. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784, 2014.
  33. In situ observation of dislocation nucleation and escape in a submicrometre aluminium single crystal. Nature materials, 8(2):95–100, 2009.
  34. Orientation mapping of semicrystalline polymers using scanning electron nanobeam diffraction. Micron, 88:30–36, 2016.
  35. LT-GAN: Self-supervised GAN with latent transformation detection. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp.  3189–3198, 2021.
  36. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  2536–2544, 2016.
  37. Deep feature transfer learning in combination with traditional features predicts survival among patients with lung adenocarcinoma. Tomography, 2:388–395, 2016.
  38. Improving language understanding by generative pre-training. OpenAI blog, 2018.
  39. ImageNet-21K pretraining for the masses. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
  40. U-Net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pp.  234–241. Springer, 2015.
  41. Instance segmentation of dislocations in TEM images. In 2023 IEEE 23rd International Conference on Nanotechnology (NANO), pp.  1–6. IEEE, 2023.
  42. Deep learning segmentation of complex features in atomic-resolution phase-contrast transmission electron microscopy images. Microscopy and microanalysis, 27(4):804–814, 2021.
  43. Angular reconstitution-based 3D reconstructions of nanomolecular structures from superresolution light-microscopy images. Proceedings of the National Academy of Sciences, 114:9273–9278, 2017.
  44. MoCo-CXR: MoCo pretraining improves representation and transferability of chest X-ray models. 2020. doi: 10.48550/arxiv.2010.05352.
  45. Deep transfer learning approaches in performance analysis of brain tumor classification using MRI images. Journal of Healthcare Engineering, 2022:1–17, 2022.
  46. Data-mining of in-situ TEM experiments: Towards understanding nanoscale fracture. Computational materials science, 216:111830, 2023.
  47. Microstructure segmentation with deep learning encoders pre-trained on a large microscopy dataset. npj Computational Materials, 8(1), September 2022.
  48. Towards foundation models for scientific machine learning: Characterizing scaling and transfer behavior. Advances in Neural Information Processing Systems, 36, 2024.
  49. Deep high-resolution representation learning for human pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  5693–5703, 2019.
  50. Understanding the influence of receptive field and network complexity in neural network-guided TEM image analysis. Microscopy and Microanalysis, 28(6):1896–1904, December 2022.
  51. GAN-based image colorization for self-supervised visual feature learning. Sensors, 22(4):1599, 2022.
  52. Transfer learning in magnetic resonance brain imaging: A systematic review. Journal of imaging, 7(4):66, 2021.
  53. Developing and evaluating deep neural network-based denoising for nanoparticle TEM images with ultra-low signal-to-noise. Microscopy and Microanalysis, 27:1431–1447, 2021.
  54. Deep high-resolution representation learning for visual recognition. IEEE Transactions on Pattern Analysis & Machine Intelligence, 43(10):3349–3364, oct 2021.
  55. SimMIM: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  9653–9663, 2022.
  56. Stare at what you see: Masked image modeling without reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  22732–22741, 2023.
  57. Data-mining of in-situ TEM experiments: On the dynamics of dislocations in cocrfemnni alloys. Acta Materialia, 241:118394, 2022a.
  58. Transfer-learning-based raman spectra identification. Journal of Raman Spectroscopy, 51(1):176–186, 2020.
  59. Contrastive spatio-temporal pretext learning for self-supervised video representation. Proceedings of the AAAI Conference on Artificial Intelligence, 36:3380–3389, 2022b.
  60. Contrastive representation learning for hand shape estimation. In DAGM German Conference on Pattern Recognition, pp. 250–264. Springer, 2021.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Lightbulb Streamline Icon: https://streamlinehq.com

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Don't miss out on important new AI/ML research

See which papers are being discussed right now on X, Reddit, and more:

Explore Trending Papers

“Emergent Mind helps me see which AI papers have caught fire online.”

Philip

Philip

Creator, AI Explained on YouTube