Develop Communication Complexity for computationally restricted agents

Develop a more sophisticated variant of Communication Complexity that explicitly models computationally restricted agents—such as agents limited to the operations available within Transformer attention layers—and use this framework to rigorously study and characterize the limitations of Transformer architectures.

Background

The paper uses classical Communication Complexity to prove limitations of single-layer multi-headed Transformers on function composition. The authors note that standard lower bounds in Communication Complexity are conservative because they allow agents with unbounded computational power and arbitrary encoding schemes, whereas in their reduction the agents’ messages are constrained to specific numeric aggregates dictated by the softmax-based attention computation.

Motivated by this mismatch, the authors call for a more refined Communication Complexity framework tailored to computationally restricted agents, which could more accurately capture and quantify the limitations of architectures like Transformers in tasks such as function evaluation and composition.

References

This observation highlights an interesting open problem, the opportunity to develop a more sophisticated variant of Communication Complexity for computationally restricted agents in order to study the limitations of devices such as the Transformer.

— On Limitations of the Transformer Architecture  (2402.08164 - Peng et al., 2024) in Discussion

We hypothesize that a restricted receiver cannot rely on the full program transcript and therefore benefits from a more self-sufficient packet representation; testing this explanation across additional tasks and successful global models remains future work.

— What You Can't See Is What You Learn: Restricted Evidence Visibility Favors Compositional Generalization in Shared-Genome Language-Model Societies  (2608.20054 - Marincat, 20 Aug 2026) in Section 5, Mechanism: value-indexed packet transplants