Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
149 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Catalytic Role Of Noise And Necessity Of Inductive Biases In The Emergence Of Compositional Communication (2111.06464v2)

Published 11 Nov 2021 in cs.LG, cs.AI, and cs.CL

Abstract: Communication is compositional if complex signals can be represented as a combination of simpler subparts. In this paper, we theoretically show that inductive biases on both the training framework and the data are needed to develop a compositional communication. Moreover, we prove that compositionality spontaneously arises in the signaling games, where agents communicate over a noisy channel. We experimentally confirm that a range of noise levels, which depends on the model and the data, indeed promotes compositionality. Finally, we provide a comprehensive study of this dependence and report results in terms of recently studied compositionality metrics: topographical similarity, conflict count, and context independence.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (53)
  1. Self-assembling Games. The British Journal for the Philosophy of Science, 68(2):329–353.
  2. Batali, J. (1998). Computational simulations of the emergence of grammar. Approach to the Evolution of Language, pages 405–426.
  3. Emergence of Communication in an Interactive World with Consistent Speakers. arXiv preprint arXiv:1809.00549.
  4. How agents see things: On visual representations in an emergent language game. Empirical Methods in Natural Language Processing.
  5. Understanding Linguistic Evolution by Visualizing the Emergence of Topographic Mappings. Artificial Life, 12(2):229–242.
  6. 3d shapes dataset. https://github.com/deepmind/3dshapes-dataset/.
  7. Compositionality and Generalization In Emergent Languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4427–4442.
  8. Compositional Obverter Communication Learning From Raw Visual Input. In International Conference on Learning Representations.
  9. Chomsky, N. (1957). Syntactic structures. Mouton de Gruyter.
  10. Emergence of Compositional Language with Deep Generational Transmission. arXiv preprint arXiv:1904.09067.
  11. Learning cooperative visual dialog agents with deep reinforcement learning. In Proceedings of the IEEE international conference on computer vision, pages 2951–2960.
  12. Connectionism and cognitive architecture: a critical analysis. Cognition, 28(1-2):3–71.
  13. Learning to Communicate with Deep Multi-Agent Reinforcement Learning. NIPS’16 Proceedings of the 30th International Conference on Neural Information Processing Systems.
  14. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pages 3259–3269.
  15. Game Theory. MIT Press.
  16. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677.
  17. Inductive bias and language expressivity in emergent communication. arXiv preprint arXiv:2012.02875.
  18. A very condensed survey and critique of multiagent deep reinforcement learning. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems, pages 2146–2148.
  19. beta-VAE: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations.
  20. Categorical reparametrization with gumbel-softmax. In Proceedings International Conference on Learning Representations 2017.
  21. Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning. In International Conference on Machine Learning, pages 3040–3049.
  22. Language as an abstraction for hierarchical deep reinforcement learning. In Advances in Neural Information Processing Systems, volume 32.
  23. Discrete autoencoders for sequence models. arXiv preprint arXiv:1801.09797.
  24. Disentangling by factorising. In International Conference on Machine Learning, pages 2649–2658.
  25. Kirby, S. (2001). Spontaneous evolution of linguistic structure-an iterated learning model of the emergence of regularity and irregularity. IEEE Transactions on Evolutionary Computation, 5(2):102–110.
  26. Developmentally motivated emergence of compositional communication via template transfer. arXiv preprint arXiv:1910.06079.
  27. Measuring non-trivial compositionality in emergent communication. arXiv preprint arXiv:2010.15058.
  28. Natural Language Does Not Emerge ’Naturally’ in Multi-Agent Dialog. arXiv preprint arXiv:1706.08502.
  29. Kriegeskorte, N. (2008). Representational similarity analysis – connecting the branches of systems neuroscience. Frontiers in Systems Neuroscience.
  30. Emergence of compositional language in communication through noisy channel. ICML 2020 Workshop LaReL.
  31. Building Machines That Learn and Think Like People. Behavioral and brain sciences, 40.
  32. Emergence of linguistic communication from referential games with symbolic and pixel input. In International Conference on Learning Representations.
  33. Multi-Agent Cooperation and the Emergence of (Natural) Language. In International Conference on Learning Representations.
  34. Emergent translation in multi-agent communication. In International Conference on Learning Representations.
  35. Levenshtein, V. I. (1966). Binary codes capable of correcting deletions, insertions, and reversals. In Soviet physics doklady, volume 10, pages 707–710. Soviet Union.
  36. Lewis, D. K. (1969). Convention: a philosophical study. Blackwell.
  37. Ease-of-teaching and language structure from emergent communication. In Advances in Neural Information Processing Systems, pages 15825–15835.
  38. Challenging common assumptions in the unsupervised learning of disentangled representations. In International Conference on Machine Learning, pages 4114–4124.
  39. Emergence of grounded compositional language in multi-agent populations. In Thirty-second AAAI conference on artificial intelligence.
  40. The evolution of language. Proceedings of the National Academy of Sciences, 96(14):8028–8033.
  41. Learning and the Emergence of Coordinated Communication. Center for Research on Language Newsletter, 11.
  42. Language Development From an Ecological Perspective: Ecologically Valid Ways to Abstract Symbols. Ecological Psychology, 30(1):39–73.
  43. Compositional languages emerge in a neural iterated learning model. In International Conference on Learning Representations.
  44. Capacity, Bandwidth, and Compositionality in Emergent Language Learning. In Autonomous Agents and Multiagent Systems, pages 1125–1133.
  45. Semantic hashing. International Journal of Approximate Reasoning, 50(7):969–978.
  46. Shannon, C. E. (1948). A mathematical theory of communication. The Bell system technical journal, 27(3):379–423.
  47. Skyrms, B. (2010). Signals: evolution, learning, & information. Oxford University Press, Oxford ; New York.
  48. Towards graph representation learning in emergent communication. arXiv preprint arXiv:2001.09063.
  49. Exploring structural inductive biases in emergent communication. arXiv preprint arXiv:2002.01335.
  50. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929–1958.
  51. Steinert-Threlkeld, S. (2020). Towards the Emergence of Non-trivial Compositionaliy. Philosophy of Science.
  52. Visualizing data using t-sne. Journal of machine learning research, 9(11).
  53. How to use t-sne effectively. Distill.
Citations (18)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now