Concept-Best-Matching: Evaluating Compositionality in Emergent Communication (2403.14705v1)
Abstract: Artificial agents that learn to communicate in order to accomplish a given task acquire communication protocols that are typically opaque to a human. A large body of work has attempted to evaluate the emergent communication via various evaluation measures, with \emph{compositionality} featuring as a prominent desired trait. However, current evaluation procedures do not directly expose the compositionality of the emergent communication. We propose a procedure to assess the compositionality of emergent communication by finding the best-match between emerged words and natural language concepts. The best-match algorithm provides both a global score and a translation-map from emergent words to natural language concepts. To the best of our knowledge, it is the first time that such direct and interpretable mapping between emergent words and human concepts is provided.
- Jacob Andreas and Dan Klein. 2016. Reasoning about pragmatics with neural listeners and speakers. In EMNLP.
- Henry Brighton and Simon Kirby. 2006. Understanding linguistic evolution by visualizing the emergence of topographic mappings. Artificial life, 12(2):229–242.
- Emergent quantized communication. arXiv preprint arXiv:2211.02412.
- Compositionality and generalization in emergent languages. In ACL, pages 4427–4442.
- Emergent communication at scale. In ICLR.
- Serhii Havrylov and Ivan Titov. 2017. Emergence of language with multi-agent games: Learning to communicate with sequences of symbols. NeurIPS, 30.
- John E Hopcroft and Richard M Karp. 1973. An n^5/2 algorithm for maximum matchings in bipartite graphs. SIAM Journal on computing, 2(4):225–231.
- Compositionality decomposed: How do neural networks generalise? Journal of Artificial Intelligence Research, 67:757–795.
- Categorical reparameterization with gumbel-softmax. In ICLR.
- EGG: a toolkit for research on emergence of lanGuage in games. In EMNLP-IJCNLP: System Demonstrations, pages 55–60.
- Harold W Kuhn. 1955. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83–97.
- Alexander Kuhnle and Ann Copestake. 2017. Shapeworld-a new test methodology for multimodal language understanding. arXiv preprint arXiv:1704.04517.
- Emergence of linguistic communication from referential games with symbolic and pixel input. In ICLR.
- Multi-agent cooperation and the emergence of (natural) language. In ICLR.
- Vladimir I Levenshtein et al. 1966. Binary codes capable of correcting deletions, insertions, and reversals. In Soviet physics doklady, volume 10, pages 707–710. Soviet Union.
- Fushan Li and Michael Bowling. 2019. Ease-of-teaching and language structure from emergent communication. NeurIPS, 32.
- Jesse Mu and Noah Goodman. 2021. Emergent communication of generalizations. Advances in Neural Information Processing Systems, 34:17994–18007.
- Noam Slonim. 2002. The information bottleneck: Theory and applications. Ph.D. thesis, Hebrew University of Jerusalem Jerusalem, Israel.
- Generalization and translatability in emergent communication via informational constraints. In NeurIPS 2022 Workshop on Information-Theoretic Principles in Cognitive Systems.
- Iterated learning for emergent systematicity in VQA. In ICLR.
- Information theoretic measures for clusterings comparison: is a correction for chance necessary? In Proceedings of the 26th annual international conference on machine learning, pages 1073–1080.
- Linking emergent and natural languages via corpus transfer. In ICLR.
- A Rate–Distortion view of human pragmatic reasoning? In Proceedings of the Society for Computation in Linguistics, pages 347–348.
- Boaz Carmeli (14 papers)
- Yonatan Belinkov (111 papers)
- Ron Meir (38 papers)