Stability of bipartite consensus in multihead self-attention dynamics

Determine whether bipartite consensus points are stable equilibria of the multihead discrete-time self-attention dynamics when the attention matrices are state-dependent and the layer matrix is symmetric with a strictly dominant positive principal eigenvalue.

Background

The paper extends its single-head discrete-time self-attention analysis to a multihead transformer model in which each head has its own attention, query, key, and value matrices, and the head outputs are combined through an output matrix. For a symmetric layer matrix with a strictly dominant positive principal eigenvalue, the paper establishes local asymptotic stability of the consensus equilibrium aligned with the principal eigenvector.

In contrast, the stability behavior of bipartite consensus equilibria—configurations in which tokens occupy two antipodal points—is not resolved for the multihead system. Although such equilibria can be locally asymptotically stable in the single-head model, the paper reports no analytical or numerical evidence that they are stable in the multihead case, leaving their stability an explicit unresolved question.