Mixed jeu de taquin and a problem of Soojin Cho

Abstract: Serrano (2010) introduced the shifted plactic monoid, governing Haiman's (1989) mixed insertion algorithm, as a type B analogue of the classical plactic monoid that connects jeu de taquin of Young tableaux with the Robinson-Schensted-Knuth insertion algorithm. Serrano proposed a corresponding definition of skew shifted plactic Schur functions. Cho (2013) disproved Serrano's conjecture regarding this definition, by showing that the functions do not live in the desired ring and hence cannot provide an algebraic interpretation of tableau rectification or of the corresponding structure coefficients. Cho asked for a new definition with particular properties. We introduce such a definition and prove that it behaves as desired. We also introduce a new jeu de taquin theory that computes mixed insertion.

• The paper introduces mixed jeu de taquin for shifted semistandard tableaux and proves that its rectification agrees with Haiman’s mixed insertion, completing a key missing link in shifted tableau theory.
• The authors define corrected skew plactic Schur P-functions using modified Sagan–Worley rectification and show that they belong to the ring generated by plactic Schur P-functions.
• The resulting expansion coefficients are governed by the generalized Sagan–Worley counts b_{λ,μ}^ν and match the coefficients of ordinary skew Schur P-functions, although mixed rectification is not confluent.

Background and motivation

In type A, the Schur functions $s_\lambda$ form a basis of the ring of symmetric functions, and their structure coefficients—the Littlewood–Richardson coefficients—admit a combinatorial interpretation through the interlocking machinery of jeu de taquin on Young tableaux, the Robinson–Schensted–Knuth (RSK) insertion algorithm, and the plactic monoid. In particular, plactic skew Schur functions encode both rectification and RSK insertion algebraically.

For the shifted setting relevant to the projective representation theory of symmetric groups and the Schubert calculus of Lagrangian and maximal orthogonal Grassmannians, the analogues of Schur functions are the Schur $P$- and $Q$-functions. Here the situation is asymmetric: Sagan–Worley insertion has an associated jeu de taquin but no known plactic monoid, while Haiman's mixed insertion has a corresponding shifted plactic monoid (introduced by Serrano) but no known jeu de taquin. Serrano proposed a definition of plactic skew Schur $P$-functions intended to play the type B role that plactic skew Schur functions play in type A; Cho disproved his conjecture by exhibiting examples showing these functions do not lie in the ring generated by plactic Schur $P$-functions, and posed the problem of finding a corrected definition whose expansion coefficients in the plactic Schur $P$-basis are "nice." This paper resolves Cho's problem and, independently, constructs the missing jeu de taquin for mixed insertion.

Mixed jeu de taquin

The authors introduce a mixed jeu de taquin on shifted semistandard tableaux. The procedure places bullets $\bullet$ in the bottom row of the inner shape and slides entries past them according to six families of local rules: diagonal slides, singular slides, non-singular slides, and three further rule collections, some of which coincide locally with Sagan–Worley slides while others behave quite differently. Entries are processed in order of "availability," using the total order $$\underline{1} < \overline{1} < \underline{2} < \overline{2} < \cdots$$, with ties broken by position. Rectification proceeds row by row until a straight shape is reached.

A central structural result is that mixed rectification is well defined: every intermediate configuration is a semistandard tableau with holes, and no low entry ever occupies a diagonal cell. The proof rests on several lemmas establishing that once an entry becomes least available it "slides completely" (never moves again), and that the order in which entries become least available respects the alphabet order. Notably, the authors state plainly that their jeu de taquin lacks confluence, distinguishing it from ordinary jeu de taquin; they note precedent for useful non-confluent theories in $K$-theoretic jeu de taquin.

The main theorem of this section shows that mixed rectification computes Haiman's mixed insertion: for any shifted tableau $T$ and high letter $P$0, performing mixed insertion of $P$1 into $P$2 yields precisely $P$3, where $P$4 places $P$5 on the antidiagonal above $P$6. The proof tracks "singular" entries—those corresponding to bumped letters during insertion—and shows by induction that the nucleus of each intermediate rectification configuration is a subtableau of the corresponding stage of the insertion process. Thus the paper supplies the jeu de taquin counterpart to the shifted plactic monoid, completing the analogy with the type A picture in which jeu de taquin and RSK are two faces of the same structure.

A solution to Cho's problem

The second contribution defines skew plactic Schur $P$7-functions via a modified Sagan–Worley rectification algorithm that admits low entries on the diagonal. The modification alters the slide rules when equal low entries meet: such entries are placed in the same column rather than compared, and diagonal configurations involving equal entries raise one copy. The definition is:

$P$8

where $P$9 denotes the set of $Q$0-tableaux (semistandard tableaux possibly with low diagonal entries), $Q$1 raises all diagonal entries, $Q$2 is the modified Sagan–Worley rectification, and $Q$3 denotes the shifted plactic class under Serrano's monoid.

The key supporting result, proved here for completeness since prior proofs were difficult to retrieve or absent from the literature, is a generalized Sagan–Worley theorem: the number of $Q$4-tableaux of shape $Q$5 rectifying to any fixed $Q$6-tableau $Q$7 of shape $Q$8 equals the coefficient $Q$9. The proof proceeds through standardization and a case analysis depending on whether the southwestmost instances of each letter are high or low, using a lemma that rectification preserves which letter is southwestmost.

The main theorem then establishes that $P$0 lies in $P$1—the ring generated by plactic Schur $P$2-functions—with expansion

$P$3

Consequently, the expansion coefficients of $P$4 in the plactic Schur $P$5-basis coincide exactly with those of the ordinary skew Schur $P$6-function in the ordinary Schur $P$7-basis. This directly answers Cho's requirements: the functions live in the correct ring and describe the multiplicities of ordinary $P$8 in a transparent way. It is worth emphasizing that the solution's essential tool is the Sagan–Worley jeu de taquin rather than the new mixed jeu de taquin—an interesting division of labor between the two type B insertion theories.

Limitations and open questions

The paper concedes several points. First, mixed jeu de taquin does not satisfy confluence, so the familiar uniqueness-of-rectification property fails; whether a confluent variant exists is not addressed. Second, the generalized Sagan–Worley counting theorem relies on results whose earlier proofs were described as difficult to retrieve, and the authors supply proofs partly for completeness—suggesting the literature record here was incomplete. Third, the authors express hope that their non-confluent jeu de taquin will lead to developments analogous to $P$9-theoretic jeu de taquin, but no such results are established here. Finally, the potential connection between mixed rectification and the structure constants of the Lagrangian Grassmannian is raised as unexplored.

Conclusion

This paper resolves Cho's open problem by defining skew plactic Schur $P$0-functions that lie in the ring generated by plactic Schur $P$1-functions and expand with coefficients matching the classical Schur $P$2-expansion, governed by the numbers $P$3. In parallel, it introduces a mixed jeu de taquin that provably computes Haiman's mixed insertion, thereby supplying the jeu de taquin side of a theory previously accessible only through the shifted plactic monoid. Together, these results restore the type A trichotomy—insertion, jeu de taquin, plactic algebra—in the shifted setting, albeit with a jeu de taquin lacking confluence.