Backward Responsibility in Transition Systems Using General Power Indices
Abstract: To improve reliability and the understanding of AI systems, there is increasing interest in the use of formal methods, e.g. model checking. Model checking tools produce a counterexample when a model does not satisfy a property. Understanding these counterexamples is critical for efficient debugging, as it allows the developer to focus on the parts of the program that caused the issue. To this end, we present a new technique that ascribes a responsibility value to each state in a transition system that does not satisfy a given safety property. The value is higher if the non-deterministic choices in a state have more power to change the outcome, given the behaviour observed in the counterexample. For this, we employ a concept from cooperative game theory -- namely general power indices, such as the Shapley value -- to compute the responsibility of the states. We present an optimistic and pessimistic version of responsibility that differ in how they treat the states that do not lie on the counterexample. We give a characterisation of optimistic responsibility that leads to an efficient algorithm for it and show computational hardness of the pessimistic version. We also present a tool to compute responsibility and show how a stochastic algorithm can be used to approximate responsibility in larger models. These methods can be deployed in the design phase, at runtime and at inspection time to gain insights on causal relations within the behavior of AI systems.
- A Game-Theoretic Account of Responsibility Allocation. In IJCAI, 1773–1779. International Joint Conferences on Artificial Intelligence Organization.
- Principles of model checking. MIT Press.
- From symptom to cause: localizing errors in counterexample traces. In Proceedings of the 30th ACM SIGPLAN-SIGACT symposium on Principles of programming languages, 97–105. Association for Computing Machinery.
- Explaining counterexamples using causality. Formal Methods in System Design, 40: 20–40.
- Bell, G. I. 2007. Solving triangular peg solitaire. arXiv preprint math/0703865.
- Temporal causality in reactive systems. In International Symposium on Automated Technology for Verification and Analysis, 208–224. Springer.
- Value Theory without Efficiency. Mathematics of Operations Research, 6(1): 122–128.
- Blameworthiness in multi-agent settings. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, 525–532.
- Automata, Logics, and Infinite Games: A Guide to Current Research [outcome of a Dagstuhl seminar, February 2001], volume 2500 of Lecture Notes in Computer Science. Springer.
- Error explanation with distance metrics. International Journal on Software Tools for Technology Transfer, 8: 229–247.
- Towards formal definitions of blameworthiness, intention, and moral responsibility. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32.
- PRISM 4.0: Verification of Probabilistic Real-time Systems. In Gopalakrishnan, G.; and Qadeer, S., eds., Proc. 23rd International Conference on Computer Aided Verification (CAV’11), volume 6806 of LNCS, 585–591. Springer.
- The PRISM Benchmark Suite. In QEST, 203–204. IEEE Computer Society.
- Shapley-Shubik and Banzhaf Indices Revisited. Mathematics of Operations Research, 26(1): 89–104.
- Lehmann, J. 2024. Tool to compute backward responsibility in transition systems using general power indices.
- Causality checking for complex system models. In International Workshop on Verification, Model Checking, and Abstract Interpretation, 248–267. Springer.
- Responsibility and verification: Importance value in temporal logics. In LICS, 1–14. IEEE.
- Fault localization with nearest neighbor queries. In 18th IEEE International Conference on Automated Software Engineering, 2003. Proceedings., 30–39. IEEE.
- Toward verified artificial intelligence. Communications of the ACM, 65(7): 46–55.
- Shapley, L. S.; et al. 1953. A value for n-person games. Contributions to the Theory of Games, 2.
- Van de Poel, I. 2011. The relation between forward-looking and backward-looking responsibility. In Moral responsibility: Beyond free will and determinism, 37–52. Springer.
- Whodunit? causal analysis for counterexamples. In International Symposium on Automated Technology for Verification and Analysis, 82–95. Springer.
- Zeller, A. 2002. Isolating cause-effect chains from computer programs. In SIGSOFT FSE, 1–10. ACM.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.