Papers
Topics
Authors
Recent
Assistant
AI Research Assistant
Well-researched responses based on relevant abstracts and paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses.
Gemini 2.5 Flash
Gemini 2.5 Flash 134 tok/s
Gemini 2.5 Pro 41 tok/s Pro
GPT-5 Medium 27 tok/s Pro
GPT-5 High 24 tok/s Pro
GPT-4o 102 tok/s Pro
Kimi K2 196 tok/s Pro
GPT OSS 120B 441 tok/s Pro
Claude Sonnet 4.5 37 tok/s Pro
2000 character limit reached

Integrating Symbolic Reasoning into Neural Generative Models for Design Generation (2310.09383v2)

Published 13 Oct 2023 in cs.AI

Abstract: Design generation requires tight integration of neural and symbolic reasoning, as good design must meet explicit user needs and honor implicit rules for aesthetics, utility, and convenience. Current automated design tools driven by neural networks produce appealing designs but cannot satisfy user specifications and utility requirements. Symbolic reasoning tools, such as constraint programming, cannot perceive low-level visual information in images or capture subtle aspects such as aesthetics. We introduce the Spatial Reasoning Integrated Generator (SPRING) for design generation. SPRING embeds a neural and symbolic integrated spatial reasoning module inside the deep generative network. The spatial reasoning module samples the set of locations of objects to be generated from a backtrack-free distribution. This distribution modifies the implicit preference distribution, which is learned by a recursive neural network to capture utility and aesthetics. Sampling from the backtrack-free distribution is accomplished by a symbolic reasoning approach, SampleSearch, which zeros out the probability of sampling spatial locations violating explicit user specifications. Embedding symbolic reasoning into neural generation guarantees that the output of SPRING satisfies user requirements. Furthermore, SPRING offers interpretability, allowing users to visualize and diagnose the generation process through the bounding boxes. SPRING is also adept at managing novel user specifications not encountered during its training, thanks to its proficiency in zero-shot constraint transfer. Quantitative evaluations and a human study reveal that SPRING outperforms baseline generative models, excelling in delivering high design quality and better meeting user specifications.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (107)
  1. Differentiable convex optimization layers. In Advances in Neural Information Processing Systems, pages 9558–9570, 2019.
  2. Learning convex optimization models. IEEE/CAA Journal of Automatica Sinica, 8(8):1355–1364, 2021.
  3. Learning convex optimization control policies. In Alexandre M. Bayen, Ali Jadbabaie, George Pappas, Pablo A. Parrilo, Benjamin Recht, Claire Tomlin, and Melanie Zeilinger, editors, Proceedings of the 2nd Conference on Learning for Dynamics and Control, volume 120 of Proceedings of Machine Learning Research, pages 361–373. PMLR, 10–11 Jun 2020.
  4. Dax: Deep argumentative explanation for neural networks. CoRR, abs/2012.05766, 2020.
  5. Thinking fast and slow with deep learning and tree search. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
  6. Conversational multi-hop reasoning with neural commonsense knowledge and symbolic logic rules. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP 2021), 2021.
  7. Sparx: Sparse argumentative explanations for neural networks. CoRR, abs/2301.09559, 2023.
  8. CLR-DRNets: Curriculum Learning with Restarts to Solve Visual Combinatorial Games. In Laurent D. Michel, editor, 27th International Conference on Principles and Practice of Constraint Programming (CP 2021), volume 210 of Leibniz International Proceedings in Informatics (LIPIcs), pages 17:1–17:14, Dagstuhl, Germany, 2021. Schloss Dagstuhl – Leibniz-Zentrum für Informatik.
  9. Entropy-based logic explanations of neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 6046–6054, 2022.
  10. Handbook of Formal Argumentation. College Publications, 2018.
  11. A note on the inception score. arXiv preprint arXiv:1801.01973, 2018.
  12. Neuron constraints to model complex real-world problems. In Principles and Practice of Constraint Programming–CP 2011: 17th International Conference, CP 2011, Perugia, Italy, September 12-16, 2011. Proceedings 17, pages 115–129. Springer, 2011.
  13. Neural-Symbolic Learning and Reasoning: A Survey and Interpretation. IOS Press, Amsterdam, 2022.
  14. Constraint acquisition. Artificial Intelligence, 244:315–342, 2017. Combining Constraint Solving with Mining and Learning.
  15. The Inductive Constraint Programming Loop. IEEE Intelligent Systems, 2018.
  16. Thinking fast and slow in ai. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 15042–15046, 2021.
  17. Improving deep learning models via constraint-based domain knowledge: a brief survey. arXiv preprint arXiv:2005.10691, 2020.
  18. Convex optimization. Cambridge university press, 2004.
  19. End-to-end object detection with transformers. CoRR, abs/2005.12872, 2020.
  20. Building-gan: Graph-conditioned architectural volumetric design generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 11956–11965, October 2021.
  21. Automating crystal-structure phase mapping by combining deep learning with constraint reasoning. Nature Machine Intelligence, 3(9):812–822, 2021.
  22. Application of ai technology in interior design. In E3S Web of Conferences, volume 179, page 02105. E3S Web of Conferences, 2020.
  23. On the properties of neural machine translation: Encoder–decoder approaches. In SSST@EMNLP, 2014.
  24. Logic explained networks. Artificial Intelligence, 314:103822, 2023.
  25. A constraint-based approach to learning and explanation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 3658–3665, 2020.
  26. Extracting dialogical explanations for review aggregations with argumentative dialogical agents. In Proc. AAMAS 2019, pages 1261–1269. IFAAMS, 2019.
  27. Data-empowered argumentation for dialectically explainable predictions. In Proc. ECAI 2020, FAIA 325, pages 2449–2456. IOS Press, 2020.
  28. O Cocarascu and F Toni. Argumentation for machine learning: A survey. In Proc. COMMA 2016, FAIA 287, pages 219–230. IOS Press, 2016.
  29. I Davidson and SS Ravi. Making existing clusterings fairer: Algorithms, complexity results and insights. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, pages 3733–3740, New York, NY, USA, 2020. AAAI Press.
  30. I Davidson and SS Ravi. Towards auditing unsupervised learning algorithms and human processes for fairness. arXiv preprint arXiv:2209.11762, 2022.
  31. Argflow: A toolkit for deep argumentative explanations for neural networks. In Proc. AAMAS 2021, pages 1761–1763. ACM, 2021.
  32. Generative scene graph networks. In International Conference on Learning Representations, 2021.
  33. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  34. Teaching the old dog new tricks: Supervised learning with constraints. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 3742–3749, 2021.
  35. Semantic image manipulation using scene graphs. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5213–5222, 2020.
  36. Relational reinforcement learning. In Inductive Logic Programming: 8th International Conference, ILP-98 Madison, Wisconsin, USA, July 22–24, 1998 Proceedings 8, pages 11–22. Springer, 1998.
  37. Genesis: Generative scene inference and sampling with object-centric latent representations. In International Conference on Learning Representations, 2020.
  38. Jonathan St BT Evans. In two minds: dual-process accounts of reasoning. Trends in cognitive sciences, 7(10):454–459, 2003.
  39. Learning explanatory rules from noisy data. JAIR, 1:11172, 2018.
  40. Fast and slow planning, 2023.
  41. Improving fairness generalization through a sample-robust optimization method. Machine Learning, Jul 2022.
  42. Improving fairness generalization through a sample-robust optimization method. Machine Learning, 112(6):2131–2192, 2023.
  43. Flexible and inherently comprehensible knowledge representation for data-efficient learning and trustworthy human-machine teaming in manufacturing environments. In Proceedings of the Neuro-symbolic AI for Agent and Multi-Agent Systems [NeSyMAS] Workshop, part of the 22nd International Conference on Autonomous Agents and Multiagent Systems, London, 2023. Presented on May 30, 2023.
  44. Thinking fast and slow in ai: The role of metacognition. In Giuseppe Nicosia, Varun Ojha, Emanuele La Malfa, Gabriele La Malfa, Panos Pardalos, Giuseppe Di Fatta, Giovanni Giuffrida, and Renato Umeton, editors, Machine Learning, Optimization, and Data Science, pages 502–509, Cham, 2023. Springer Nature Switzerland.
  45. Unconditional scene graph generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16362–16371, 2021.
  46. Marco Gori. Logic explained networks. In 17th International Workshop on Neural-Symbolic Learning and Reasoning (NESY 2023), Certosa di Pontignano, Siena, Italy, 2023.
  47. Deep Residual Learning for Image Recognition. In Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR ’16, pages 770–778. IEEE, June 2016.
  48. A hybrid neural network model for commonsense reasoning. In Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing, pages 13–21, Hong Kong, China, November 2019. Association for Computational Linguistics.
  49. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
  50. Case-based reasoning and neural network based expert system for personalization. Expert Systems with Applications, 32(1):77–85, 2007.
  51. Herbert Jaeger. Conceptors: an easy introduction. ArXiv, abs/1406.2671, 2014.
  52. Constraint reasoning embedded structured prediction. Journal of Machine Learning Research, 23(345):1–40, 2022.
  53. Image generation from scene graphs. In CVPR, 2018.
  54. Daniel Kahneman. Thinking, fast and slow. Farrar, Straus and Giroux, New York, 2011.
  55. Henry Kautz. The third ai summer: Aaai robert s. engelmore memorial lecture. AI Magazine, 43(1):105–125, 2022.
  56. Accelerating chip design with machine learning. IEEE Micro, 40(6):23–32, 2020.
  57. Learning combinatorial optimization algorithms over graphs. Advances in neural information processing systems, 30, 2017.
  58. A deep learning approach for interior designing of apartment building architecture using u2 net. In 2022 International Conference on Sustainable Computing and Data Communication Systems (ICSCDS), pages 1465–1471, 2022.
  59. Graph neural networks meet neural-symbolic computing: A survey and perspective. In Christian Bessiere, editor, Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 4877–4884. International Joint Conferences on Artificial Intelligence Organization, 7 2020. Survey track.
  60. Yann LeCun. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review, 62, 2022.
  61. Visual-attention gan for interior sketch colourisation. IET Image Processing, 15(4):997–1007, 2021.
  62. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer, 2014.
  63. The different role of working memory in open-ended versus closed-ended creative problem solving: A dual-process theory account. Creativity Research Journal, 25(1):85–96, 2013.
  64. Learning to compose visual relations. Advances in Neural Information Processing Systems, 34:23166–23178, 2021.
  65. Boosting combinatorial problem modeling with machine learning. arXiv preprint arXiv:1807.05517, 2018.
  66. The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. In International Conference on Learning Representations, 2019.
  67. Glancenets: Interpretable, leak-proof concept-based models. In Advances in Neural Information Processing Systems, 2022.
  68. Detect, understand, act: A neuro-symbolic hierarchical reinforcement learning framework. Machine Learning, 111(4):1523–1549, 2022.
  69. Decision focused learning for prediction + optimisation problems. In Proceedings of AAAI, Constraint Programming and Machine Learning Bridge Program, 2023.
  70. House-gan: Relational generative adversarial networks for graph-constrained house layout generation. In European Conference on Computer Vision, pages 162–177. Springer, 2020.
  71. House-gan++: Generative adversarial layout refinement network towards intelligent computational agent for professional architects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13632–13641, 2021.
  72. Glide: Towards photorealistic image generation and editing with text-guided diffusion models, 2021.
  73. Improving coherence and consistency in neural sequence models with dual-system, neuro-symbolic reasoning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 25192–25204. Curran Associates, Inc., 2021.
  74. Towards neural network-based reasoning. ArXiv, abs/1508.05508, 2015.
  75. Relational reasoning using neural networks: A survey. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 29(Supp02):237–258, 2021.
  76. Classifier-based constraint acquisition. Annals of Mathematics and Artificial Intelligence, 89:655–674, 2021.
  77. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
  78. Zero-shot text-to-image generation. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8821–8831. PMLR, 18–24 Jul 2021.
  79. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
  80. Using constraint-based operators to solve the vehicle routing problem with time windows. Journal of heuristics, 8:43–58, 2002.
  81. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
  82. A simple neural network module for relational reasoning. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
  83. Deep unsupervised learning using nonequilibrium thermodynamics. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 2256–2265, Lille, France, 07–09 Jul 2015. PMLR.
  84. The shifting sands of creative thinking: Connections to dual-process theory. Thinking & Reasoning, 21(1):40–60, 2015.
  85. Neural qbafs: Explaining neural networks under lrp-based argumentation frameworks. In Proc. AIxIA 2021, LNCS 13196, pages 429–444. Springer, 2021.
  86. Neuro-symbolic program search for autonomous driving decision module design. In Jens Kober, Fabio Ramos, and Claire Tomlin, editors, Proceedings of the 2020 Conference on Robot Learning, volume 155 of Proceedings of Machine Learning Research, pages 21–30. PMLR, 16–18 Nov 2021.
  87. Object-centric image generation from layouts. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 2647–2655, 2021.
  88. Alexandru Telea. An image inpainting technique based on the fast marching method. Journal of graphics tools, 9(1):23–34, 2004.
  89. Francesca Toni. A roadmap for neuro-argumentative learning. In 17th International Workshop on Neural-Symbolic Learning and Reasoning (NESY 2023), Certosa di Pontignano, Siena, Italy, 2023. Short paper.
  90. A simple method for commonsense reasoning, 2018.
  91. Learning constraint models from data. In Proceedings of AAAI, Constraint Programming and Machine Learning Bridge Program, 2023.
  92. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  93. Biosystems design by machine learning. ACS synthetic biology, 9(7):1514–1533, 2020.
  94. Learning mdps from features: Predict-then-optimize for sequential decision making by reinforcement learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 8795–8806. Curran Associates, Inc., 2021.
  95. Rlayout: Interior design system based on reinforcement learning. In 2019 12th International Symposium on Computational Intelligence and Design (ISCID), volume 1, pages 117–120, 2019.
  96. A learning algorithm for continually running fully recurrent neural networks. Neural Computation, 1(2):270–280, 1989.
  97. Spring: Gpt-4 out-performs rl algorithms by studying papers and reasoning, 2023.
  98. Deep structured generative models. arXiv preprint arXiv:1807.03877, 2018.
  99. Embedding decision diagrams into generative adversarial networks. In International Conference on Integration of Constraint Programming, Artificial Intelligence, and Operations Research, pages 616–632. Springer, 2019.
  100. Safe reinforcement learning via probabilistic logic shields. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence (IJCAI-23). IJCAI, 2023.
  101. Neural-symbolic vqa: Disentangling reasoning from vision and language understanding. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
  102. Deep Learning Architect: Classification for Architectural Design Through the Eye of Artificial Intelligence, pages 249–265. Springer International Publishing, Cham, 2019.
  103. Relational deep reinforcement learning. arXiv preprint arXiv:1806.01830, 2018.
  104. Learning visual commonsense for robust scene graph generation. In European Conference on Computer Vision, pages 642–657. Springer, 2020.
  105. Adding conditional control to text-to-image diffusion models, 2023.
  106. Image generation from layout. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8584–8593, 2019.
  107. Argumentative xai: A survey. In Zhi-Hua Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 4392–4399. International Joint Conferences on Artificial Intelligence Organization, 8 2021. Survey Track.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
Lightbulb Streamline Icon: https://streamlinehq.com

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up