Structure of connected nested automorphism groups

Abstract: A nested group is an increasing union of a sequence of algebraic groups. In this paper, we describe maximal nested unipotent subgroups of $\mathrm{Aut}(X)$, where $X$ is an affine variety. It turns out that they are similar to the group of triangular automorphisms of $\mathbb{A}^n$. We show that if an abstract subgroup of $\mathrm{Aut}(X)$ consists of unipotent elements, then it is closed if and only if it is nested. This implies that a connected nested subgroup of $\mathrm{Aut}(X)$ is closed, answering a question of Kraft and Zaidenberg (2022, arXiv:2203.11356). We also extend the recent description of maximal commutative unipotent subgroups by Regeta and van Santen (2024, arXiv:2112.04784), offering a direct construction method and relating them to our description.