Minimal generating and separating sets for O(3)-invariants of several matrices

Abstract: Given an algebra $F[H]^G$ of polynomial invariants of an action of the group $G$ over the vector space $H$, a subset $S$ of $F[H]^G$ is called separating if $S$ separates all orbits that can be separated by $F[H]^G$. A minimal separating set is found for some algebras of matrix invariants of several matrices over an infinite field of arbitrary characteristic different from two in case of the orthogonal group. Namely, we consider the following cases: 1) $GL(3)$-invariants of two matrices; 2) $O(3)$-invariants of $d>0$ skew-symmetric matrices; 3) $O(4)$-invariants of two skew-symmetric matrices; 4) $O(3)$-invariants of two symmetric matrices. A minimal generating set is also given for the algebra of orthogonal invariants of three $3\times 3$ symmetric matrices.