Oligomorphic groups, their automorphism groups, and the complexity of their isomorphism

Abstract: The paper follows two interconnected directions. 1. Let $G$ be a Roelcke precompact closed subgroup of the group $\Sym(\omega)$ of permutations of the natural numbers. Then $\Inn(G)$ is closed in $\Aut(G)$, where $\Aut(G)$ carries the topology of pointwise convergence for its (faithful) action on the cosets of open subgroups. Under the stronger hypothesis that~$G$ is oligomorphic, $N_G/G$ is profinite, where $N_G$ denotes the normaliser of~$G$ in $\Sym(\omega)$, and the topological group $\Out(G)= \Aut(G)/\Inn(G)$ is totally disconnected, locally compact. 2a. We provide a general method to show smoothness of the isomorphism relation for appropriate Borel classes of oligomorphic groups. We apply it to two such classes: the oligomorphic groups with no algebraicity, and the oligomorphic groups with finitely many {essential} subgroups up to conjugacy. 2b. Using this method we also show that if $G$ is in such a Borel class, then $\Aut(G)$ is topologically isomorphic to an oligomorphic group, and $\Out(G)$ is profinite.