On hyperinvariant subspaces of operators containing unilateral shifts

Abstract: The hyperinvariant subspace problem for Hilbert space operators $T$ containing a unilateral shift is addressed. The discussion is based on a similarity model of $T$, which is an operator-matrix $\widehat T= [T_{i,j}]<em>3$ where $T</em>{1,1}$ is the simple unilateral shift $S$ and $T_{3,3}$ is a cyclic diagonal operator $D$. The existence of $D$ is established by the technique resulting almost invariant half-spaces in \cite{APTT}; see also \cite{Tc} and \cite{HP}. For any operator $Q=[Q_{i,j}]<em>3$ in the commutant of $\widehat T$, the entry $Q</em>{3,1}$ intertwines $S$ with $D$ up to a transformation of rank at most 1. These entries form a linear manifold ${\cal L}<em>{3,1}$. We focus on 3-dimensional cross-sections of ${\cal L}</em>{3,1}$. These are subspaces of $3\times 3$ complex matrices, transformed into singular matrices by a canonical mapping. If such a subspace ${\cal L}$ is not transitive, then $T$ has a nontrivial hyperinvariant subspace. A throrough study reveals that ${\cal L}$ can be transitive only if it has a very specific basis. Consequences of the existence of nontrivial hyperinvariant subspaces in the presence of shift-type invariant subspaces are also discussed.

• The paper establishes a reduction theorem linking shift-type invariant subspaces to the existence of hyperinvariant subspaces via block-matrix similarity.
• It employs explicit analysis of commutant cross-sections and detailed algebraic classification to characterize transitivity among D-singular matrix subspaces.
• The work advances the hyperinvariant subspace problem by providing rigorous criteria that constrain counterexamples and guide further research in operator theory.

Hyperinvariant Subspaces for Operators Containing Unilateral Shifts

Problem Context and Motivation

The invariant and hyperinvariant subspace problems (ISP, HSP) for Hilbert space operators are among the most persistent open problems in operator theory. For a bounded linear operator $T$ on a (separable, infinite-dimensional) Hilbert space, the ISP asks whether $T$ must always possess a nontrivial invariant subspace, while the HSP inquires whether every nonscalar $T$ admits a hyperinvariant subspace (invariant under every operator commuting with $T$). While the answers are negative in the general Banach space setting, the Hilbert space case remains unsettled.

This work focuses on a natural class of operators—those containing a shift-type invariant subspace, i.e., admitting a restriction similar to the unilateral shift, $S$. The presence of such a highly nontrivial restriction is a strong symmetry property. The central question is: Does any Hilbert space operator with a shift-type invariant subspace necessarily possess a nontrivial hyperinvariant subspace?

The theoretical motivation includes not only resolving HSP/ISP for a broad operator class but also elucidating the structure of commutants and their finite-dimensional "cross-sections" via novel linear-algebraic techniques.

Similarity Model and Commutant Cross-Sections

A key step is to model an operator $T$ with a shift-type invariant subspace via a block-matrix similarity:

$$T \sim \begin{bmatrix} S & * & * \ 0 & * & * \ 0 & F_* & D_* \end{bmatrix}$$

Here, $S$ is the unilateral shift; $D_*$ is a cyclic diagonal operator constructed via almost-invariant half-space techniques; $F_*$ is rank one. This similarity allows explicit analysis of $T$0's commutant.

Operators in the commutant, written $T$1, satisfy that $T$2 intertwines $T$3 with $T$4 up to a rank-one perturbation: $T$5

The set of such $T$6 forms a linear manifold. The focus is placed on 3-dimensional compressions ("cross-sections") of this manifold, specifically as subspaces of $T$7 matrices.

A distinguishing property investigated is transitivity: a subspace of matrices is transitive if for any nonzero vector $T$8, its range under the subspace is the whole space (i.e., there are no invariant subspaces). If a relevant cross-section fails to be transitive, one can construct a nontrivial hyperinvariant subspace for $T$9.

Singular, Transitive Subspaces of Matrices: Fine Structure

The paper undertakes a systematic study of cross-section subspaces in $T$0 that, under a canonical transformation incorporating a cyclic diagonal matrix, map into the variety of singular matrices. These "D-singular" subspaces are characterized as follows:

• Their dimension is at most $T$1.
• They are potentially transitive only for dimension $T$2 or $T$3.
• For $T$4-dimensional subspaces, transitivity and $T$5-singularity are incompatible; no such matrix subspace is both.
• For $T$6-dimensional subspaces, exhaustive algebraic analysis shows that, except for a very special canonical form (essentially one algebraic exceptional case per orbit), $T$7-singularity implies lack of transitivity.

The technical approach involves:

• Introducing canonical bases for subspaces annihilating the third column,
• Expressing the $T$8-singular property in terms of vanishing of certain determinant polynomials,
• Parameterizing all exceptional cases by detailed pattern analysis.

The only configurations where a $T$9-dimensional, $T$0-singular subspace can also be transitive are when the basis takes a canonical block-diagonal form (E7), with explicit algebraic dependencies among parameters.

Implications for the Hyperinvariant Subspace Problem

The upshot is a reduction theorem: To demonstrate the existence of hyperinvariant subspaces for Hilbert space operators containing an S-type shift, it suffices to analyze the case where, for every integer $T$1, the associated cross-sections of the commutant take a specified canonical pattern. When even one cross-section deviates from this pattern or falls outside the five-dimensional case, the operator has a nontrivial hyperinvariant subspace.

As a corollary, for asymptotically nonvanishing absolutely continuous contractions (those for which all orbits do not tend to zero), if the residual set of the unitary asymptote fills the unit circle (i.e., the system is "maximal" in a spectral sense), the operator must have such a subspace.

For the further-specialized class of cyclic, absolutely continuous polynomially bounded operators with bilateral shift as unitary asymptote, nontrivial hyperinvariant subspaces are also guaranteed.

Practical and Theoretical Impact

These results constitute a major step in clarifying the landscape of the hyperinvariant subspace problem for highly non-normal operators. The methods introduced—especially the use of almost-invariant half-spaces, commutant block analysis, and cross-sectional algebra—are poised to have applications beyond the shift-based setting, potentially affecting the analysis of non-selfadjoint operator algebras, transitive operator subspaces, and the structure of commuting tuples.

On the practical side, these findings sharpen our understanding of when commutants admit tractable invariant structures, influencing both spectral theory and applications relying on decomposability of operator actions (e.g., in quantum theory and signal processing).

Prospective directions include:

• Extending to higher-dimensional shift components and multivariate shift analogues,
• Systematizing the exceptional canonical forms for dimension $T$2,
• Investigating analogous properties in non-Hilbertian or more general Banach space contexts.

Conclusion

This paper establishes a firm connection between the presence of shift-type invariant subspaces and the existence of nontrivial hyperinvariant subspaces for Hilbert space operators, modulo the detailed structure of certain finite-dimensional commutant cross-sections. It provides exhaustive algebraic classification criteria and a reduction scheme that narrows remaining open cases to highly structured, explicitly described settings. This significantly constrains possible counterexamples and advances both the operator-theoretic and linear-algebraic understanding of the hyperinvariant subspace problem.