Algorithmic decidability of Engel's property for automaton groups

Abstract: We consider decidability problems associated with Engel's identity ($[\cdots[[x,y],y],\dots,y]=1$ for a long enough commutator sequence) in groups generated by an automaton. We give a partial algorithm that decides, given $x,y$, whether an Engel identity is satisfied. It succeeds, importantly, in proving that Grigorchuk's $2$-group is not Engel. We consider next the problem of recognizing Engel elements, namely elements $y$ such that the map $x\mapsto[x,y]$ attracts to ${1}$. Although this problem seems intractable in general, we prove that it is decidable for Grigorchuk's group: Engel elements are precisely those of order at most $2$. Our computations were implemented using the package FR within the computer algebra system GAP.