Papers
Topics
Authors
Recent
Search
2000 character limit reached

Inferring Capabilities from Task Performance with Bayesian Triangulation

Published 21 Sep 2023 in cs.AI | (2309.11975v1)

Abstract: As machine learning models become more general, we need to characterise them in richer, more meaningful ways. We describe a method to infer the cognitive profile of a system from diverse experimental data. To do so, we introduce measurement layouts that model how task-instance features interact with system capabilities to affect performance. These features must be triangulated in complex ways to be able to infer capabilities from non-populational data -- a challenge for traditional psychometric and inferential tools. Using the Bayesian probabilistic programming library PyMC, we infer different cognitive profiles for agents in two scenarios: 68 actual contestants in the AnimalAI Olympics and 30 synthetic agents for O-PIAAGETS, an object permanence battery. We showcase the potential for capability-oriented evaluation.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (54)
  1. An Introduction to MCMC for Machine Learning. Machine Learning, 50(1-2): 5–43.
  2. How do infants reason about physical events?
  3. Ben-Gal, I. 2008. Bayesian Networks. John Wiley & Sons, Ltd. ISBN 9780470061572.
  4. The person and the variable in developmental psychology. Zeitschrift für Psychologie/Journal of Psychology.
  5. Hamiltonian Monte Carlo for Hierarchical Models. arXiv 1312.0906.
  6. Brier, G. W. 1950. Verification of Forecasts Expressed in Terms of Probability. Monthly Weather Review, 78(1): 1.
  7. Not a Number: Identifying Instance Features for Capability-Oriented Evaluation. IJCAI.
  8. Rethink reporting of evaluation results in AI. Science, 380(6641): 136–138.
  9. Intuitive physical reasoning about occluded objects by inexperienced chicks. Proceedings of the Royal Society B: Biological Sciences, 278(1718): 2621–2627. Publisher: Royal Society.
  10. Cooper, G. F. 1990. The computational complexity of probabilistic inference using bayesian belief networks. Artificial Intelligence, 42(2): 393–405.
  11. The Animal-AI Testbed and Competition. Proceedings of Machine Learning Research, (123): 164–176.
  12. De Ayala, R. J. 2009. Theory and practice of item response theory. Guilford Publications.
  13. Item response theory for psychologists. L. Erlbaum.
  14. A temporal same-object advantage in the tunnel effect: facilitated change detection for persisting objects. Journal of Experimental Psychology: Human Perception and Performance, 32(4): 840.
  15. Google. 2017. Google Colaboratory. https://colab.research.google.com/. Accessed 2023-05-23.
  16. Individual differences in cognitive functions. In D. C. Berliner & R. C. Calfee (Eds.), Handbook of educational psychology (pp. 186–242). Prentice Hall International.
  17. Human-like social skills in dogs? Trends in cognitive sciences, 9(9): 439–444.
  18. Hastings, W. K. 1970. Monte Carlo Sampling Methods Using Markov Chains and Their Applications. Biometrika, 57(1): 97–109.
  19. Vindicating methodological triangulation. Synthese, 196(8): 3067–3081.
  20. Humans have evolved specialized skills of social cognition: The cultural intelligence hypothesis. Science, 317(5843): 1360–1366.
  21. The No-U-Turn Sampler: Adaptively Setting Path Lengths in Hamiltonian Monte Carlo. Journal of Machine Learning Research, 15(47): 1593–1623.
  22. Dimensions of commonsense knowledge. arXiv preprint arXiv:2101.04640.
  23. Adversarial examples for evaluating reading comprehension systems. arXiv preprint arXiv:1707.07328.
  24. Unity: A General Platform for Intelligent Agents. CoRR, abs/1809.02627.
  25. Cattell–Horn–Carroll abilities and cognitive tests: What we’ve learned from 20 years of research. Psychology in the Schools, 47(7): 635–650.
  26. Dynabench: Rethinking benchmarking in NLP. arXiv preprint arXiv:2104.14337.
  27. Reduced, reused and recycled: The life of a dataset in machine learning research. arXiv preprint arXiv:2112.01716.
  28. Building machines that learn and think like people. Behavioral and brain sciences, 40.
  29. Lee, M. D. 2011. How cognitive modeling can benefit from hierarchical Bayesian models. Journal of Mathematical Psychology, 55(1): 1–7.
  30. Letteri, C. A. 1980. Cognitive profile: Basic determinant of academic achievement. The Journal of Educational Research, 73(4): 195–199.
  31. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110.
  32. Software testing for machine learning. In AAAI, volume 34, 13576–13582.
  33. Variable-and person-centered approaches to examining temperament vulnerability and resilience to the effects of contextual risk. Journal of research in personality, 67: 61–74.
  34. Murphy, A. H. 1973. A New Vector Partition of the Probability Score. Journal of Applied Meteorology and Climatology, 12(4): 595 – 600.
  35. Murphy, K. P. 2023. Probabilistic Machine Learning: Advanced Topics. MIT Press.
  36. Adversarial NLI: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599.
  37. Osband, I.; et al. 2019. Behaviour Suite for Reinforcement Learning. arXiv preprint arXiv:1908.03568.
  38. AI and the Everything in the Whole Wide World Benchmark. In Vanschoren, J.; and Yeung, S., eds., Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1. Curran.
  39. Reckase, M. D. 2009. Multidimensional item response theory models. In Multidimensional item response theory, 79–112. Springer.
  40. Beyond accuracy: Behavioral testing of NLP models with CheckList. ACL2020, arXiv preprint arXiv:2005.04118.
  41. Increasing Generality in Machine Learning through Procedural Content Generation. Nature Machine Intelligence, 428–436.
  42. Diversify your datasets: Analyzing generalization via controlled variance in adversarial datasets. arXiv preprint arXiv:1910.09302.
  43. Probabilistic programming in Python using PyMC3. PeerJ Computer Science, 2: e55.
  44. Scholl, B. J. 2007. Object persistence in philosophy and psychology. Mind & Language, 22(5): 563–591.
  45. Artificial intelligence and the common sense of animals. Trends in cognitive sciences, 24(11): 862–872.
  46. Cognitive test batteries in animal cognition research: evaluating the past, present and future of comparative psychometrics. Animal cognition, 20(6): 1003–1018.
  47. Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615.
  48. Stanford, P. K. 2006. Exceeding our grasp: Science, history, and the problem of unconceived alternatives, volume 1. Oxford University Press.
  49. Benchmarking Machine Reading Comprehension: A Psychological Perspective. arXiv preprint arXiv:2004.01912.
  50. Structural Equation Modeling, chapter 23. John Wiley & Sons, Ltd. ISBN 9781118133880.
  51. Assessment in infancy: Ordinal scales of psychological development.
  52. Evaluating object permanence in embodied agents using the animal-AI environment. EBeM’22: Workshop on AI Evaluation Beyond Metrics, July 25, 2022, Vienna, Austria.
  53. Swag: A large-scale adversarial dataset for grounded commonsense inference. arXiv preprint arXiv:1808.05326.
  54. HellaSwag: Can a Machine Really Finish Your Sentence? arXiv preprint arXiv:1905.07830.
Citations (6)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up