Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

A Survey on Verification and Validation, Testing and Evaluations of Neurosymbolic Artificial Intelligence (2401.03188v2)

Published 6 Jan 2024 in cs.AI

Abstract: Neurosymbolic AI is an emerging branch of AI that combines the strengths of symbolic AI and sub-symbolic AI. A major drawback of sub-symbolic AI is that it acts as a "black box", meaning that predictions are difficult to explain, making the testing & evaluation (T&E) and validation & verification (V&V) processes of a system that uses sub-symbolic AI a challenge. Since neurosymbolic AI combines the advantages of both symbolic and sub-symbolic AI, this survey explores how neurosymbolic applications can ease the V&V process. This survey considers two taxonomies of neurosymbolic AI, evaluates them, and analyzes which algorithms are commonly used as the symbolic and sub-symbolic components in current applications. Additionally, an overview of current techniques for the T&E and V&V processes of these components is provided. Furthermore, it is investigated how the symbolic part is used for T&E and V&V purposes in current neurosymbolic applications. Our research shows that neurosymbolic AI as great potential to ease the T&E and V&V processes of sub-symbolic AI by leveraging the possibilities of symbolic AI. Additionally, the applicability of current T&E and V&V methods to neurosymbolic AI is assessed, and how different neurosymbolic architectures can impact these methods is explored. It is found that current T&E and V&V techniques are partly sufficient to test, evaluate, verify, or validate the symbolic and sub-symbolic part of neurosymbolic applications independently, while some of them use approaches where current T&E and V&V methods are not applicable by default, and adjustments or even new approaches are needed. Our research shows that there is great potential in using symbolic AI to test, evaluate, verify, or validate the predictions of a sub-symbolic model, making neurosymbolic AI an interesting research direction for safe, secure, and trustworthy AI.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (120)
  1. A. d. Garcez and L. C. Lamb, “Neurosymbolic ai: The 3 rd wave,” Artificial Intelligence Review, pp. 1–20, 2023.
  2. C. M. Bishop, “Neural networks and their applications,” Review of scientific instruments, vol. 65, no. 6, pp. 1803–1832, 1994.
  3. M. Garnelo and M. Shanahan, “Reconciling deep learning with symbolic artificial intelligence: representing objects and relations,” Current Opinion in Behavioral Sciences, vol. 29, pp. 17–23, 2019.
  4. K. Acharya, W. Raza, C. Dourado, A. Velasquez, and H. H. Song, “Neurosymbolic reinforcement learning and planning: A survey,” IEEE Transactions on Artificial Intelligence, pp. 1–14, 2023.
  5. A. Velasquez, “Transfer from imprecise and abstract models to autonomous technologies (tiamat),” Defense Advanced Research Projects Agency (DARPA) Program Solicitation, 2023.
  6. Oct 2023. [Online]. Available: https://www.whitehouse.gov/briefing-room/presidential-actions/2023/10/30/executive-order-on-the-safe-secure-and-trustworthy-development-and-use-of-artificial-intelligence/
  7. X. Huang, “Safety and reliability of deep learning: (brief overview),” in Proceedings of the 1st International Workshop on Verification of Autonomous & Robotic Systems, ser. VARS ’21.   New York, NY, USA: Association for Computing Machinery, 2021. [Online]. Available: https://doi.org/10.1145/3459086.3459636
  8. X. Huang, D. Kroening, W. Ruan, J. Sharp, Y. Sun, E. Thamo, M. Wu, and X. Yi, “A survey of safety and trustworthiness of deep neural networks: Verification, testing, adversarial attack and defence, and interpretability,” Computer Science Review, vol. 37, p. 100270, 2020. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1574013719302527
  9. D. Wallace and R. Fujii, “Software verification and validation: Its role in computer assurance and its relationship with software project management standards,” 1989-09-05 1989. [Online]. Available: https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=905731
  10. R. G. Sargent, “A tutorial on validation and verification of simulation models,” in Proceedings of the 20th conference on Winter simulation, 1988, pp. 33–39.
  11. D. Wallace and R. Fujii, “Software verification and validation: an overview,” IEEE Software, vol. 6, no. 3, pp. 10–17, 1989.
  12. H. Kautz, “The third ai summer: Aaai robert s. engelmore memorial lecture,” AI Magazine, vol. 43, no. 1, pp. 93–104, Mar. 2022. [Online]. Available: https://ojs.aaai.org/index.php/aimagazine/article/view/19122
  13. D. Yu, B. Yang, D. Liu, H. Wang, and S. Pan, “A survey on neural-symbolic learning systems,” Neural Networks, vol. 166, pp. 105–126, 2023. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0893608023003398
  14. J. M. Zhang, M. Harman, L. Ma, and Y. Liu, “Machine learning testing: Survey, landscapes and horizons,” IEEE Transactions on Software Engineering, vol. 48, no. 01, pp. 1–36, 2022.
  15. H. B. Braiek and F. Khomh, “On testing machine learning programs,” Journal of Systems and Software, vol. 164, p. 110542, 2020. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0164121220300248
  16. R. C. Cardoso, G. Kourtis, L. A. Dennis, C. Dixon, M. Farrell, M. Fisher, and M. Webster, “A review of verification and validation for space autonomous systems,” Current Robotics Reports, vol. 2, no. 3, pp. 273–283, 2021.
  17. A. d. Garcez, S. Bader, H. Bowman, L. C. Lamb, L. de Penning, B. Illuminoo, H. Poon, and C. G. Zaverucha, “Neural-symbolic learning and reasoning: A survey and interpretation,” Neuro-Symbolic Artificial Intelligence: The State of the Art, vol. 342, no. 1, p. 327, 2022.
  18. A. Sheth, K. Roy, and M. Gaur, “Neurosymbolic artificial intelligence (why, what, and how),” IEEE Intelligent Systems, vol. 38, no. 3, pp. 56–62, 2023.
  19. M. Gaur, K. Gunaratna, S. Bhatt, and A. Sheth, “Knowledge-infused learning: A sweet spot in neuro-symbolic ai,” IEEE Internet Computing, vol. 26, no. 4, pp. 5–11, 2022.
  20. M. K. Sarker, L. Zhou, A. Eberhart, and P. Hitzler, “Neuro-symbolic artificial intelligence,” AI Communications, vol. 34, no. 3, pp. 197–209, 2021.
  21. Z. Susskind, B. Arden, L. K. John, P. Stockton, and E. B. John, “Neuro-symbolic ai: An emerging class of ai workloads and their characterization,” arXiv preprint arXiv:2109.06133, 2021.
  22. W. Wang and Y. Yang, “Towards data-and knowledge-driven artificial intelligence: A survey on neuro-symbolic computing,” arXiv preprint arXiv:2210.15889, 2022.
  23. W. Gibaut, L. Pereira, F. Grassiotto, A. Osorio, E. Gadioli, A. Munoz, S. Gomes, and C. d. Santos, “Neurosymbolic ai and its taxonomy: a survey,” arXiv preprint arXiv:2305.08876, 2023.
  24. L. N. DeLong, R. F. Mir, M. Whyte, Z. Ji, and J. D. Fleuriot, “Neurosymbolic ai for reasoning on graph structures: A survey,” arXiv preprint arXiv:2302.07200, 2023.
  25. L. N. DeLong, R. F. Mir, Z. Ji, F. N. C. Smith, and J. D. Fleuriot, “Neurosymbolic ai for reasoning on biomedical knowledge graphs,” arXiv preprint arXiv:2307.08411, 2023.
  26. K. Hamilton, A. Nayak, B. Božić, and L. Longo, “Is neuro-symbolic ai meeting its promises in natural language processing? a structured review,” Semantic Web, no. Preprint, pp. 1–42, 2022.
  27. K. W. Church, “Word2vec,” Natural Language Engineering, vol. 23, no. 1, pp. 155–162, 2017.
  28. J. Pennington, R. Socher, and C. D. Manning, “Glove: Global vectors for word representation,” in Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), 2014, pp. 1532–1543.
  29. D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton et al., “Mastering the game of go without human knowledge,” nature, vol. 550, no. 7676, pp. 354–359, 2017.
  30. J. Mao, C. Gan, P. Kohli, J. B. Tenenbaum, and J. Wu, “The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision,” CoRR, vol. abs/1904.12584, 2019. [Online]. Available: http://arxiv.org/abs/1904.12584
  31. R. Manhaeve, S. Dumancic, A. Kimmig, T. Demeester, and L. De Raedt, “Deepproblog: Neural probabilistic logic programming,” Advances in neural information processing systems, vol. 31, 2018.
  32. G. Lample and F. Charton, “Deep learning for symbolic mathematics,” CoRR, vol. abs/1912.01412, 2019. [Online]. Available: http://arxiv.org/abs/1912.01412
  33. L. Serafini, I. Donadello, and A. d. Garcez, “Learning and reasoning in logic tensor networks: Theory and application to semantic image interpretation,” in Proceedings of the Symposium on Applied Computing, ser. SAC ’17.   New York, NY, USA: Association for Computing Machinery, 2017, p. 125–130. [Online]. Available: https://doi.org/10.1145/3019612.3019642
  34. P. Smolensky, M. Lee, X. He, W.-t. Yih, J. Gao, and L. Deng, “Basic reasoning with tensor product representations,” 2016. [Online]. Available: https://arxiv.org/abs/1601.02745
  35. M. Alshiekh, R. Bloem, R. Ehlers, B. Könighofer, S. Niekum, and U. Topcu, “Safe reinforcement learning via shielding,” in Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, ser. AAAI’18/IAAI’18/EAAI’18.   AAAI Press, 2018.
  36. M. Alshiekh, R. Bloem, R. Ehlers, B. Könighofer, S. Niekum, and U. Topcu, “Safe reinforcement learning via shielding,” arXiv preprint arXiv:1708.08611, 2017.
  37. I. H. Anellis, “Peirce’s truth-functional analysis and the origin of the truth table,” History and Philosophy of Logic, vol. 33, no. 1, pp. 87–97, 2012.
  38. W. Schwarz, “GitHub - wo/tpg: Tree Proof Generator — github.com,” https://github.com/wo/tpg, [Accessed 01-Dec-2022].
  39. E. Huaman, E. Kärle, and D. Fensel, “Knowledge graph validation,” 2020. [Online]. Available: https://arxiv.org/abs/2005.01389
  40. Z. H. Syed, N. Srivastava, M. Röder, and A.-C. N. Ngomo, “Copaal-an interface for explaining facts using corroborative paths.” in ISWC (Satellites), 2019, pp. 201–204.
  41. J. Lehmann, D. Gerber, M. Morsey, and A.-C. Ngonga Ngomo, “Defacto-deep fact validation,” in International semantic web conference.   Springer, 2012, pp. 312–327.
  42. A. Rula, M. Palmonari, S. Rubinacci, A.-C. Ngonga Ngomo, J. Lehmann, A. Maurino, and D. Esteves, “Tisco: Temporal scoping of facts,” in Companion Proceedings of The 2019 World Wide Web Conference, 2019, pp. 959–960.
  43. Z. H. Syed, M. Röder, and A.-C. Ngonga Ngomo, “Factcheck: Validating rdf triples using textual evidence,” in Proceedings of the 27th ACM International Conference on Information and Knowledge Management, 2018, pp. 1599–1602.
  44. G. Ercan, S. Elbassuoni, and K. Hose, “Retrieving textual evidence for knowledge graph facts,” in European Semantic Web Conference.   Springer, 2019, pp. 52–67.
  45. R. Speck and A.-C. N. Ngomo, “Leopard—a baseline approach to attribute prediction and validation for knowledge graph population,” Journal of Web Semantics, vol. 55, pp. 102–107, 2019.
  46. A. Padia, F. Ferraro, and T. Finin, “Surface: semantically rich fact validation with explanations,” arXiv preprint arXiv:1810.13223, 2018.
  47. S. Metzger, S. Elbassuoni, K. Hose, and R. Schenkel, “S3k: seeking statement-supporting top-k witnesses,” in Proceedings of the 20th ACM international conference on Information and knowledge management, 2011, pp. 37–46.
  48. T. Rebele, F. Suchanek, J. Hoffart, J. Biega, E. Kuzey, and G. Weikum, “Yago: A multilingual knowledge base from wikipedia, wordnet, and geonames,” in The Semantic Web–ISWC 2016: 15th International Semantic Web Conference, Kobe, Japan, October 17–21, 2016, Proceedings, Part II 15.   Springer, 2016, pp. 177–185.
  49. R. Speer, J. Chin, and C. Havasi, “Conceptnet 5.5: An open multilingual graph of general knowledge,” in Proceedings of the AAAI conference on artificial intelligence, vol. 31, no. 1, 2017.
  50. D. Ferrucci, E. Brown, J. Chu-Carroll, J. Fan, D. Gondek, A. A. Kalyanpur, A. Lally, J. W. Murdock, E. Nyberg, J. Prager et al., “Building watson: An overview of the deepqa project,” AI magazine, vol. 31, no. 3, pp. 59–79, 2010.
  51. G. A. Miller, “Wordnet: A lexical database for english,” Commun. ACM, vol. 38, no. 11, p. 39–41, nov 1995. [Online]. Available: https://doi.org/10.1145/219717.219748
  52. N. Drenkow, N. Sani, I. Shpitser, and M. Unberath, “A systematic review of robustness in deep learning for computer vision: Mind the gap?” 2022.
  53. M. Wicker, X. Huang, and M. Kwiatkowska, “Feature-guided black-box safety testing of deep neural networks,” in Tools and Algorithms for the Construction and Analysis of Systems: 24th International Conference, TACAS 2018, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2018, Thessaloniki, Greece, April 14-20, 2018, Proceedings, Part I 24.   Springer, 2018, pp. 408–426.
  54. W. Ruan, X. Huang, and M. Kwiatkowska, “Reachability analysis of deep neural networks with provable guarantees,” in Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18.   International Joint Conferences on Artificial Intelligence Organization, 7 2018, pp. 2651–2659. [Online]. Available: https://doi.org/10.24963/ijcai.2018/368
  55. M. Wu, M. Wicker, W. Ruan, X. Huang, and M. Kwiatkowska, “A game-based approximate verification of deep neural networks with provable guarantees,” Theoretical Computer Science, vol. 807, pp. 298–329, 2020, in memory of Maurice Nivat, a founding father of Theoretical Computer Science - Part II. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0304397519304426
  56. G. Katz, C. Barrett, D. L. Dill, K. Julian, and M. J. Kochenderfer, “Reluplex: An efficient smt solver for verifying deep neural networks,” in Computer Aided Verification: 29th International Conference, CAV 2017, Heidelberg, Germany, July 24-28, 2017, Proceedings, Part I 30.   Springer, 2017, pp. 97–117.
  57. R. Ehlers, “Formal verification of piece-wise linear feed-forward neural networks,” in Automated Technology for Verification and Analysis: 15th International Symposium, ATVA 2017, Pune, India, October 3–6, 2017, Proceedings 15.   Springer, 2017, pp. 269–286.
  58. R. R. Bunel, I. Turkaslan, P. Torr, P. Kohli, and P. K. Mudigonda, “A unified view of piecewise linear neural network verification,” Advances in Neural Information Processing Systems, vol. 31, 2018.
  59. A. Lomuscio and L. Maganti, “An approach to reachability analysis for feed-forward relu neural networks,” arXiv preprint arXiv:1706.07351, 2017.
  60. W. Xiang, H.-D. Tran, and T. T. Johnson, “Output reachable set estimation and verification for multilayer neural networks,” IEEE Transactions on Neural Networks and Learning Systems, vol. 29, no. 11, pp. 5777–5783, 2018.
  61. C.-H. Cheng, G. Nührenberg, and H. Ruess, “Maximum resilience of artificial neural networks,” in Automated Technology for Verification and Analysis: 15th International Symposium, ATVA 2017, Pune, India, October 3–6, 2017, Proceedings 15.   Springer, 2017, pp. 251–268.
  62. N. Narodytska, S. Kasiviswanathan, L. Ryzhyk, M. Sagiv, and T. Walsh, “Verifying properties of binarized deep neural networks,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32, no. 1, 2018.
  63. N. Narodytska, “Formal analysis of deep binarized neural networks,” in Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18.   International Joint Conferences on Artificial Intelligence Organization, 7 2018, pp. 5692–5696. [Online]. Available: https://doi.org/10.24963/ijcai.2018/811
  64. T. Gehr, M. Mirman, D. Drachsler-Cohen, P. Tsankov, S. Chaudhuri, and M. Vechev, “Ai2: Safety and robustness certification of neural networks with abstract interpretation [paper presentation],” in IEEE Symposium on Security and Privacy (SP), San Francisco, CA, United States. https://doi. org/10.1109/SP, 2018.
  65. S. Wang, K. Pei, J. Whitehouse, J. Yang, and S. Jana, “Formal security analysis of neural networks using symbolic intervals,” in 27th USENIX Security Symposium (USENIX Security 18), 2018, pp. 1599–1614.
  66. A. Raghunathan, J. Steinhardt, and P. Liang, “Certified defenses against adversarial examples,” in International Conference on Learning Representations, 2018.
  67. W. Ruan, M. Wu, Y. Sun, X. Huang, D. Kroening, and M. Kwiatkowska, “Global robustness evaluation of deep neural networks with provable guarantees for the hamming distance,” in Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19.   International Joint Conferences on Artificial Intelligence Organization, 7 2019, pp. 5944–5952. [Online]. Available: https://doi.org/10.24963/ijcai.2019/824
  68. X. Huang, M. Kwiatkowska, S. Wang, and M. Wu, “Safety verification of deep neural networks,” in Computer Aided Verification: 29th International Conference, CAV 2017, Heidelberg, Germany, July 24-28, 2017, Proceedings, Part I 30.   Springer, 2017, pp. 3–29.
  69. S. Dutta, S. Jha, S. Sanakaranarayanan, and A. Tiwari, “Output range analysis for deep neural networks,” arXiv preprint arXiv:1709.09130, 2017.
  70. L. Pulina and A. Tacchella, “An abstraction-refinement approach to verification of artificial neural networks,” in Computer Aided Verification: 22nd International Conference, CAV 2010, Edinburgh, UK, July 15-19, 2010. Proceedings 22.   Springer, 2010, pp. 243–257.
  71. E. Wong and Z. Kolter, “Provable defenses against adversarial examples via the convex outer adversarial polytope,” in International conference on machine learning.   PMLR, 2018, pp. 5286–5295.
  72. M. Mirman, T. Gehr, and M. Vechev, “Differentiable abstract interpretation for provably robust neural networks,” in International Conference on Machine Learning.   PMLR, 2018, pp. 3578–3586.
  73. R. Kohavi et al., “A study of cross-validation and bootstrap for accuracy estimation and model selection,” in Ijcai, vol. 14, no. 2.   Montreal, Canada, 1995, pp. 1137–1145.
  74. N. Japkowicz, “Why question machine learning evaluation methods,” in AAAI workshop on evaluation methods for machine learning, 2006, pp. 6–11.
  75. M. Vartak, J. M. F. da Trindade, S. Madden, and M. Zaharia, “Mistique: A system to store and query model intermediates for model diagnosis,” in Proceedings of the 2018 International Conference on Management of Data, ser. SIGMOD ’18.   New York, NY, USA: Association for Computing Machinery, 2018, p. 1285–1300. [Online]. Available: https://doi.org/10.1145/3183713.3196934
  76. S. Krishnan and E. Wu, “Palm: Machine learning explanations for iterative debugging,” in Proceedings of the 2nd Workshop on Human-In-the-Loop Data Analytics, ser. HILDA ’17.   New York, NY, USA: Association for Computing Machinery, 2017. [Online]. Available: https://doi.org/10.1145/3077257.3077271
  77. J. Ding, X. Kang, and X.-H. Hu, “Validating a deep learning framework by metamorphic testing,” in Proceedings of the 2nd International Workshop on Metamorphic Testing, ser. MET ’17.   IEEE Press, 2017, p. 28–34.
  78. S. Nakajima, “Generalized oracle for testing machine learning computer programs,” in Software Engineering and Formal Methods: SEFM 2017 Collocated Workshops: DataMod, FAACS, MSE, CoSim-CPS, and FOCLASA, Trento, Italy, September 4-5, 2017, Revised Selected Papers 15.   Springer, 2018, pp. 174–179.
  79. A. Dwarakanath, M. Ahuja, S. Sikand, R. M. Rao, R. J. C. Bose, N. Dubash, and S. Podder, “Identifying implementation bugs in machine learning based image classifiers using metamorphic testing,” in Proceedings of the 27th ACM SIGSOFT international symposium on software testing and analysis, 2018, pp. 118–128.
  80. D. Pesu, Z. Q. Zhou, J. Zhen, and D. Towey, “A monte carlo method for metamorphic testing of machine translation services,” in Proceedings of the 3rd International Workshop on Metamorphic Testing, ser. MET ’18.   New York, NY, USA: Association for Computing Machinery, 2018, p. 38–45. [Online]. Available: https://doi.org/10.1145/3193977.3193980
  81. E. Breck, S. Cai, E. Nielsen, M. Salib, and D. Sculley, “The ml test score: A rubric for ml production readiness and technical debt reduction,” in 2017 IEEE International Conference on Big Data (Big Data).   IEEE, 2017, pp. 1123–1132.
  82. S. Ma, Y. Liu, W.-C. Lee, X. Zhang, and A. Grama, “Mode: automated neural network model debugging via state differential analysis and input selection,” in Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2018, pp. 175–186.
  83. N. Hynes, D. Sculley, and M. Terry, “The data linter: Lightweight, automated sanity checking for ml data sets,” in NIPS MLSys Workshop, vol. 1, no. 5, 2017.
  84. S. Schelter, D. Lange, P. Schmidt, M. Celikel, F. Biessmann, and A. Grafberger, “Automating large-scale data quality verification,” Proceedings of the VLDB Endowment, vol. 11, no. 12, pp. 1781–1794, 2018.
  85. K. Pei, Y. Cao, J. Yang, and S. Jana, “Deepxplore: Automated whitebox testing of deep learning systems,” in proceedings of the 26th Symposium on Operating Systems Principles, 2017, pp. 1–18.
  86. X. Xie, L. Ma, F. Juefei-Xu, M. Xue, H. Chen, Y. Liu, J. Zhao, B. Li, J. Yin, and S. See, “Deephunter: a coverage-guided fuzz testing framework for deep neural networks,” in Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, 2019, pp. 146–157.
  87. J. Guo, Y. Zhao, H. Song, and Y. Jiang, “Coverage guided differential adversarial testing of deep learning systems,” IEEE Transactions on Network Science and Engineering, vol. 8, no. 2, pp. 933–942, 2020.
  88. D. Meng and H. Chen, “Magnet: A two-pronged defense against adversarial examples,” in Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, ser. CCS ’17.   New York, NY, USA: Association for Computing Machinery, 2017, p. 135–147. [Online]. Available: https://doi.org/10.1145/3133956.3134057
  89. W. Tan, J. Renkhoff, A. Velasquez, Z. Wang, L. Li, J. Wang, S. Niu, F. Yang, Y. Liu, and H. Song, “Noisecam: Explainable ai for the boundary between noise and adversarial attacks,” arXiv preprint arXiv:2303.06151, 2023.
  90. S. Qiu, Q. Liu, S. Zhou, and W. Huang, “Adversarial attack and defense technologies in natural language processing: A survey,” Neurocomputing, vol. 492, pp. 278–307, 2022. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0925231222003861
  91. N. Martins, J. M. Cruz, T. Cruz, and P. Henriques Abreu, “Adversarial machine learning applied to intrusion and malware scenarios: A systematic review,” IEEE Access, vol. 8, pp. 35 403–35 419, 2020.
  92. O. Biran and C. Cotton, “Explanation and justification in machine learning: A survey,” in IJCAI-17 workshop on explainable AI (XAI), vol. 8, no. 1, 2017, pp. 8–13.
  93. S. Ruiter and N. D. De Graaf, “National context, religiosity, and volunteering: Results from 53 countries,” American Sociological Review, vol. 71, no. 2, pp. 191–210, 2006.
  94. R. D. Cook, “Detection of influential observation in linear regression,” Technometrics, vol. 19, no. 1, pp. 15–18, 1977.
  95. G. Anderson, A. Verma, I. Dillig, and S. Chaudhuri, “Neurosymbolic reinforcement learning with formally verified exploration,” Advances in neural information processing systems, vol. 33, pp. 6172–6183, 2020.
  96. T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” arXiv preprint arXiv:1509.02971, 2015.
  97. J. Achiam, D. Held, A. Tamar, and P. Abbeel, “Constrained policy optimization,” in International conference on machine learning.   PMLR, 2017, pp. 22–31.
  98. S. Gu, L. Yang, Y. Du, G. Chen, F. Walter, J. Wang, Y. Yang, and A. Knoll, “A review of safe reinforcement learning: Methods, theory and applications,” arXiv preprint arXiv:2205.10330, 2022.
  99. O. Bastani, Y. Pu, and A. Solar-Lezama, “Verifiable reinforcement learning via policy extraction,” in Advances in Neural Information Processing Systems, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, Eds., vol. 31.   Curran Associates, Inc., 2018. [Online]. Available: https://proceedings.neurips.cc/paper/2018/file/e6d8545daa42d5ced125a4bf747b3688-Paper.pdf
  100. Y. Tian, A. Luo, X. Sun, K. Ellis, W. T. Freeman, J. B. Tenenbaum, and J. Wu, “Learning to infer and execute 3d shape programs,” arXiv preprint arXiv:1901.02875, 2019.
  101. D. Trivedi, J. Zhang, S.-H. Sun, and J. J. Lim, “Learning to synthesize programs as interpretable and generalizable policies,” Advances in neural information processing systems, vol. 34, pp. 25 146–25 163, 2021.
  102. J. Huang, Z. Li, D. Jacobs, M. Naik, and S.-N. Lim, “Laser: Neuro-symbolic learning of semantic video representations,” arXiv preprint arXiv:2304.07647, 2023.
  103. X. Wang, Y. Ye, and A. Gupta, “Zero-shot recognition via semantic embeddings and knowledge graphs,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  104. A. Carlson, J. Betteridge, B. Kisiel, B. Settles, E. R. Hruschka, and T. M. Mitchell, “Toward an architecture for never-ending language learning,” in Twenty-Fourth AAAI conference on artificial intelligence, 2010.
  105. X. Chen, A. Shrivastava, and A. Gupta, “Neil: Extracting visual knowledge from web data,” in Proceedings of the IEEE international conference on computer vision, 2013, pp. 1409–1416.
  106. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein et al., “Imagenet large scale visual recognition challenge,” International journal of computer vision, vol. 115, no. 3, pp. 211–252, 2015.
  107. D. Zügner and S. Günnemann, “Certifiable robustness and robust training for graph convolutional networks,” in Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2019, pp. 246–256.
  108. V. P. Dwivedi, C. K. Joshi, A. T. Luu, T. Laurent, Y. Bengio, and X. Bresson, “Benchmarking graph neural networks,” Journal of Machine Learning Research, vol. 24, no. 43, pp. 1–48, 2023.
  109. V. Fung, J. Zhang, E. Juarez, and B. G. Sumpter, “Benchmarking graph neural networks for materials chemistry,” npj Computational Materials, vol. 7, no. 1, p. 84, 2021.
  110. N. Metropolis and S. Ulam, “The monte carlo method,” Journal of the American statistical association, vol. 44, no. 247, pp. 335–341, 1949.
  111. R. Manhaeve, S. Dumancic, A. Kimmig, T. Demeester, and L. De Raedt, “Deepproblog: Neural probabilistic logic programming,” Advances in Neural Information Processing Systems, vol. 31, 2018.
  112. A. D. Gordon, T. A. Henzinger, A. V. Nori, and S. K. Rajamani, “Probabilistic programming,” in Future of Software Engineering Proceedings, 2014, pp. 167–181.
  113. J. Guo, Y. Jiang, Y. Zhao, Q. Chen, and J. Sun, “Dlfuzz: Differential fuzzing testing of deep learning systems,” in Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2018, pp. 739–743.
  114. M. Bošnjak, T. Rocktäschel, J. Naradowsky, and S. Riedel, “Programming with a differentiable forth interpreter,” in International conference on machine learning.   PMLR, 2017, pp. 547–556.
  115. A. Odena, C. Olsson, D. Andersen, and I. Goodfellow, “Tensorfuzz: Debugging neural networks with coverage-guided fuzzing,” in International Conference on Machine Learning.   PMLR, 2019, pp. 4901–4911.
  116. A. Darwiche, “Sdd: A new canonical representation of propositional knowledge bases,” in Twenty-Second International Joint Conference on Artificial Intelligence, 2011.
  117. E. Kern, L. M. Hilty, A. Guldner, Y. V. Maksimov, A. Filler, J. Gröger, and S. Naumann, “Sustainable software products—towards assessment criteria for resource and energy efficiency,” Future Generation Computer Systems, vol. 86, pp. 199–210, 2018. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0167739X17314188
  118. A. Guldner and J. Murach, “Measuring and assessing the resource and energy efficiency of artificial intelligence of things devices and algorithms,” in Advances and New Trends in Environmental Informatics: Environmental Informatics and the UN Sustainable Development Goals.   Springer, 2022, pp. 185–199.
  119. N. Ngu, N. Lee, and P. Shakarian, “Diversity measures: Domain-independent proxies for failure in language model queries,” 2023.
  120. Z. Li, J. Huang, and M. Naik, “Scallop: A language for neurosymbolic programming,” Proceedings of the ACM on Programming Languages, vol. 7, no. PLDI, pp. 1463–1487, 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Justus Renkhoff (3 papers)
  2. Ke Feng (20 papers)
  3. Marc Meier-Doernberg (1 paper)
  4. Alvaro Velasquez (56 papers)
  5. Houbing Herbert Song (14 papers)
Citations (6)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now