On the second fundamental theorem of invariant theory for the orthosymplectic supergroup

Abstract: The first fundamental theorem of invariant theory for the orthosymplectic supergroup ${\rm OSp}(V)$ (where $V$ has superdimension $(m|2n)$) in the endomorphism algebra setting states that there is a surjective algebra homomorphism $F_r^r: B_r(m-2n)\rightarrow {\rm{End}}{{\rm OSp}(V)}(V^{\otimes r})$ from the Brauer algebra of degree $r$ to the endomorphism algebra of $V^{\otimes r}$ over ${\rm OSp}(V)$. The second fundamental theorem in this setting seeks to describe ${\rm Ker} F_r^r$ as a $2$-sided ideal of $B_r(m-2n)$. We show that ${\rm Ker} F_r^r\neq 0$ if and only if $r\geq r_c:=(m+1)(n+1)$, and present a basis and a dimension formulae for ${\rm Ker} F_r^r$. As a 2-sided ideal, ${\rm Ker} F_r^r$ for any $r\ge r_c$ is generated by ${\rm Ker} F{r_c}^{r_c}$, for which a set of generators is explicitly constructed in terms of Brauer diagrams. As applications of these results, we obtain the necessary and sufficient conditions for the endomorphism algebra $\rm{End}_{{\mathfrak {osp}}(V)}(V^{\otimes r})$ over the orthosymplectic Lie superalgebra ${\mathfrak {osp}}(V)$ to be isomorphic to $B_r(m-2n)$, and give new proofs for the main theorems in papers of G. Lehrer and R. Zhang on the second fundamental theorem of invariant theory for the orthogonal and symplectic groups.