Generalized $q,t$-Catalan numbers

Abstract: Recent work of the first author, Negut and Rasmussen, and of Oblomkov and Rozansky in the context of Khovanov--Rozansky knot homology produces a family of polynomials in $q$ and $t$ labeled by integer sequences. These polynomials can be expressed as equivariant Euler characteristics of certain line bundles on flag Hilbert schemes. The $q,t$-Catalan numbers and their rational analogues are special cases of this construction. In this paper, we give a purely combinatorial treatment of these polynomials and show that in many cases they have nonnegative integer coefficients. For sequences of length at most 4, we prove that these coefficients enumerate subdiagrams in a certain fixed Young diagram and give an explicit symmetric chain decomposition of the set of such diagrams. This strengthens results of Lee, Li and Loehr for $(4,n)$ rational $q,t$-Catalan numbers.