A generalization of a 1998 unimodality conjecture of Reiner and Stanton

Abstract: An interesting, and still wide open, conjecture of Reiner and Stanton predicts that certain "strange" symmetric differences of $q$-binomial coefficients are always nonnegative and unimodal. We extend their conjecture to a broader, and perhaps more natural, framework, by conjecturing that, for each $k\ge 5$, the polynomials $$f(k,m,b)(q)=\binom{m}{k}_q-q<sup>{\frac{k(m-b)}{2}+b-2k+2}\cdot\binom{b}{k-2}_q$$ are nonnegative and unimodal for all $m\gg_k 0$ and $b\le \frac{km-4k+4}{k-2}$ such that $kb\equiv km$ (mod 2), with the only exception of $b=\frac{km-4k+2}{k-2}$ when this is an integer. Using the KOH theorem, we combinatorially show the case $k=5$. In fact, we completely characterize the nonnegativity and unimodality of $f(k,m,b)$ for $k\le 5$. (This also provides an isolated counterexample to Reiner-Stanton's conjecture when $k=3$.) Further, we prove that, for each $k$ and $m$, it suffices to show our conjecture for the largest $2k-6$ values of $b$.