Refined $\mathrm{SU}(3)$ Vafa-Witten invariants and modularity

Abstract: We conjecture a formula for the refined $\mathrm{SU}(3)$ Vafa-Witten invariants of any smooth surface $S$ satisfying $H_1(S,\mathbb{Z}) = 0$ and $p_g(S)>0$. The unrefined formula corrects a proposal by Labastida-Lozano and involves unexpected algebraic expressions in modular functions. We prove that our formula satisfies a refined $S$-duality modularity transformation. We provide evidence for our formula by calculating virtual $\chi_y$-genera of moduli spaces of rank 3 stable sheaves on $S$ in examples using Mochizuki's formula. Further evidence is based on the recent definition of refined $\mathrm{SU}(r)$ Vafa-Witten invariants by Maulik-Thomas and subsequent calculations on nested Hilbert schemes by Thomas (rank 2) and Laarakker (rank 3).