Frequency-Time Diffusion with Neural Cellular Automata
Abstract: Despite considerable success, large Denoising Diffusion Models (DDMs) with UNet backbone pose practical challenges, particularly on limited hardware and in processing gigapixel images. To address these limitations, we introduce two Neural Cellular Automata (NCA)-based DDMs: Diff-NCA and FourierDiff-NCA. Capitalizing on the local communication capabilities of NCA, Diff-NCA significantly reduces the parameter counts of NCA-based DDMs. Integrating Fourier-based diffusion enables global communication early in the diffusion process. This feature is particularly valuable in synthesizing complex images with important global features, such as the CelebA dataset. We demonstrate that even a 331k parameter Diff-NCA can generate 512x512 pathology slices, while FourierDiff-NCA (1.1m parameters) reaches a three times lower FID score of 43.86, compared to the four times bigger UNet (3.94m parameters) with a score of 128.2. Additionally, FourierDiff-NCA can perform diverse tasks such as super-resolution, out-of-distribution image synthesis, and inpainting without explicit training.
- Structured crowdsourcing enables convolutional segmentation of histology images. Bioinformatics, 35(18): 3461–3467.
- Demystifying mmd gans. arXiv preprint arXiv:1801.01401.
- Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34: 8780–8794.
- Gilpin, W. 2019. Cellular automata as convolutional neural networks. Physical Review E, 100(3): 032402.
- Generative adversarial nets. Advances in neural information processing systems, 27.
- Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30.
- Denoising diffusion probabilistic models. Advances in neural information processing systems, 33: 6840–6851.
- Cascaded diffusion models for high fidelity image generation. The Journal of Machine Learning Research, 23(1): 2249–2281.
- Med-NCA: Robust and Lightweight Segmentation with Neural Cellular Automata. In International Conference on Information Processing in Medical Imaging, 705–716. Springer.
- Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
- Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114.
- Sdm: Spatial diffusion model for large hole image inpainting. arXiv preprint arXiv:2212.02963.
- Deep Learning Face Attributes in the Wild. In Proceedings of International Conference on Computer Vision (ICCV).
- Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 11461–11471.
- Texture generation with neural cellular automata. arXiv preprint arXiv:2105.07299.
- Growing neural cellular automata. Distill, 5(2): e23.
- Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, 8162–8171. PMLR.
- Generative adversarial neural cellular automata. arXiv preprint arXiv:2108.04328.
- DyNCA: Real-time Dynamic Texture Synthesis Using Neural Cellular Automata. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 20742–20751.
- Variational neural cellular automata. arXiv preprint arXiv:2201.12360.
- Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32.
- High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 10684–10695.
- U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, 234–241. Springer.
- Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4): 4713–4726.
- Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, 2256–2265. PMLR.
- Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502.
- Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, 1–9.
- Attention is all you need. Advances in neural information processing systems, 30.
Whiteboard
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.