Splittings and automorphisms of relatively hyperbolic groups

Abstract: We study automorphisms of a relatively hyperbolic group G. When G is one-ended, we describe Out(G) using a preferred JSJ tree over subgroups that are virtually cyclic or parabolic. In particular, when G is toral relatively hyperbolic, Out(G) is virtually built out of mapping class groups and subgroups of GL_n(Z) fixing certain basis elements. When more general parabolic groups are allowed, these subgroups of GL_n(Z) have to be replaced by McCool groups: automorphisms of parabolic groups acting trivially (i.e. by conjugation) on certain subgroups. Given a malnormal quasiconvex subgroup P of a hyperbolic group G, we view G as hyperbolic relative to P and we apply the previous analysis to describe the group Out(P to G) of automorphisms of P that extend to G: it is virtually a McCool group. If Out(P to G) is infinite, then P is a vertex group in a splitting of G. If P is torsion-free, then Out(P to G) is of type VF, in particular finitely presented. We also determine when Out(G) is infinite, for G relatively hyperbolic. The interesting case is when G is infinitely-ended and has torsion. When G is hyperbolic, we show that Out(G) is infinite if and only if G splits over a maximal virtually cyclic subgroup with infinite center. In general we show that infiniteness of Out(G) comes from the existence of a splitting with infinitely many twists, or having a vertex group that is maximal parabolic with infinitely many automorphisms acting trivially on incident edge groups.