Representations of cohomological Hall algebras and Donaldson-Thomas theory with classical structure groups

Abstract: We introduce a new class of representations of the cohomological Hall algebras of Kontsevich and Soibelman, which we call cohomological Hall modules, or CoHM for short. These representations are constructed from self-dual representations of a quiver with contravariant involution $\sigma$ and provide a mathematical model for the space of BPS states in orientifold string theory. We use the CoHM to define a generalization of the cohomological Donaldson-Thomas theory of quivers which allows the quiver representations to have orthogonal and symplectic structure groups. The associated invariants are called orientifold Donaldson-Thomas invariants. We prove the integrality conjecture for orientifold Donaldson-Thomas invariants of $\sigma$-symmetric quivers. We also formulate precise conjectures regarding the geometric meaning of these invariants and the freeness of the CoHM of a $\sigma$-symmetric quiver. We prove the freeness conjecture for disjoint union quivers, loop quivers and the affine Dynkin quiver of type $\widetilde{A}_1$. We also verify the geometric conjecture in a number of examples. Finally, we describe the CoHM of finite type quivers by constructing explicit Poincar\'{e}-Birkhoff-Witt type bases of these representations.