Negative-mass instability of the spin and motion of an atomic gas driven by optical cavity backaction

Abstract: We realize a spin-orbit interaction between the collective spin precession and center-of-mass motion of a trapped ultracold atomic gas, mediated by spin- and position-dependent dispersive coupling to a driven optical cavity. The collective spin, precessing near its highest-energy state in an applied magnetic field, can be approximated as a negative-mass harmonic oscillator. When the Larmor precession and mechanical motion are nearly resonant, cavity mediated coupling leads to a negative-mass instability, driving exponential growth of a correlated mode of the hybrid system. We observe this growth imprinted on modulations of the cavity field and estimate the full covariance of the resulting two-mode state by observing its transient decay during subsequent free evolution.