Develop Communication Complexity for computationally restricted agents

Develop a more sophisticated variant of Communication Complexity that explicitly models computationally restricted agents—such as agents limited to the operations available within Transformer attention layers—and use this framework to rigorously study and characterize the limitations of Transformer architectures.

Background

The paper uses classical Communication Complexity to prove limitations of single-layer multi-headed Transformers on function composition. The authors note that standard lower bounds in Communication Complexity are conservative because they allow agents with unbounded computational power and arbitrary encoding schemes, whereas in their reduction the agents’ messages are constrained to specific numeric aggregates dictated by the softmax-based attention computation.

Motivated by this mismatch, the authors call for a more refined Communication Complexity framework tailored to computationally restricted agents, which could more accurately capture and quantify the limitations of architectures like Transformers in tasks such as function evaluation and composition.