Hybrid convolutional–Transformer architectures for spatial reasoning

Develop hybrid architectures that combine the inductive biases of convolutional layers with the scalability of high-complexity Transformers for diffusion-based spatial reasoning tasks.

Background

The paper uses an attention-based U-Net as the diffusion reasoner and reports that preliminary experiments with the Diffusion Transformer architecture were difficult to optimize for the spatial reasoning tasks considered. The unresolved direction is to design a hybrid backbone that preserves the spatial inductive biases and optimization behavior of convolutional layers while benefiting from the scalability of high-complexity Transformer architectures. Such an architecture is intended to improve the applicability of diffusion-based reasoning models to more demanding spatial tasks without sacrificing the structural advantages of convolutional processing.

References

We leave the development of hybrid architectures, leveraging the inductive biases of convolutional layers alongside the scalability of high-complexity Transformers, for future work.

— Relational Abstractions for Spatial Reasoning with Diffusion Models  (2610.09780 - Ezquerro et al., 7 Oct 2026) in Appendix, Section 2.3, “Configuration of the Reasoner” (immediately following the discussion of preliminary Diffusion Transformer experiments)