Degenerate Whittaker Models in Representation Theory

• Degenerate Whittaker models are generalized Whittaker models defined by using characters that vanish on some root subgroups, linking representations to nilpotent coadjoint orbits.
• They provide a framework for analyzing the structure of representations by connecting the Whittaker support with the wave-front set in both real and p-adic contexts.
• Their applications span automorphic forms, Lie superalgebras, and mathematical physics, offering combinatorial, analytic, and geometric insights into modern representation theory.

Degenerate Whittaker models generalize the classical (nondegenerate) Whittaker model by allowing the defining character on a maximal unipotent subgroup to be trivial on some (possibly many) root subgroups, and they play a central role in contemporary representation theory, the analysis of automorphic forms, and areas connecting Lie theory, algebraic geometry, and mathematical physics. A degenerate Whittaker model is typically attached to a nilpotent coadjoint orbit, offering a framework to paper representations that do not admit nontrivial “generic” Whittaker functionals. The theory of degenerate Whittaker models is now understood to have deep geometric, categorical, and analytic underpinnings, reflected in their connection to nilpotent orbits, the structure of associate varieties, the branching of representation categories, and explicit analytic or combinatorial decompositions.

1. Definition and Algebraic Structure

Let $G$ be a reductive group over a local or global field with Lie algebra $\mathfrak{g}$. The classical (nondegenerate) Whittaker model for a representation $\pi$ is defined using a fixed generic character $\psi$ of the nilpotent radical $N$ of a Borel subgroup $B = NA$, yielding the space of Whittaker functionals $\operatorname{Hom}_N(\pi, \psi)$. In the degenerate case, one replaces $\psi$ with a possibly degenerate character, trivial on some root subgroups, so that $\psi$ factors through $N/E$, with $E$ a suitable subgroup.

Given a nilpotent coadjoint orbit $\mathcal{O} \subset \mathfrak{g}^*$, a degenerate Whittaker model $W_{\mathcal{O}}$ is constructed by forming a representation of $G$ induced from a character attached to a representative $y \in \mathcal{O}$. The construction typically involves:

• A Whittaker pair $(S, y)$, with $y$ nilpotent and $S$ a semisimple element giving a parabolic $P$ with unipotent radical $U$.
• The model $W_{y}$ is defined as an (oscillator) induction from a subgroup determined by the symplectic form $\omega_y(x, x') = \langle y, [x, x'] \rangle$ on the Lie algebra, leading to a Heisenberg group structure when $y$ is not regular (Gourevitch et al., 2018).
• For a representation $\pi$, one defines the "degenerate Whittaker quotient" $$\pi_{\mathcal{O}} := (W_{\mathcal{O}} \otimes \pi)_G$$.

The classical result by Kostant for semisimple Lie algebras states that the nonzero Whittaker model exists for those representations whose associated variety is the full nilpotent cone, and Matumoto extended this to less generic situations involving degenerate functionals (Gourevitch et al., 2012). The dimension of the space of Whittaker functionals (degenerate or not) is a fundamental invariant related to the geometry of the underlying orbit and the representation.

2. Characterization and Geometric Interpretation

Degenerate Whittaker models are fundamentally linked to the structure of nilpotent orbits, closure relations, and associated varieties. For a representation $\pi$, the set of all nilpotent orbits $\mathcal{O}$ such that $W_{\mathcal{O}}$ includes $\pi$ defines its "Whittaker support" $WS(\pi)$. In broad generality, the maximal elements of $WS(\pi)$ coincide with the wave-front set $WF(\pi)$ of $\pi$ (Gourevitch et al., 2018).

For real or $p$-adic groups:

• The existence of a nontrivial degenerate Whittaker model for a given orbit $\mathcal{O}$ implies that $\mathcal{O}$ is in the closure of $WF(\pi)$ (Gourevitch et al., 2012, Gourevitch et al., 2018, Gomez et al., 2015).
• In the setting of $GL_n(F)$, a smooth irreducible admissible representation $\pi$ has a degenerate Whittaker model associated to $\mathcal{O}$ if and only if $\mathcal{O}$ lies in the closure of $WF(\pi)$ (Gomez et al., 2015).

The underlying geometry is encoded in relations such as the projection of the associated variety under the map $g^* \rightarrow \mathfrak{n}^*$, and the existence of functionals is controlled by the support of the coinvariant module $C(\pi) = \pi / [N, N]\pi$. These constructions are independent of choices up to conjugacy (Gourevitch et al., 2018).

3. Explicit Realizations and Structural Results

Degenerate Whittaker models have been given precise structural descriptions in numerous settings:

• For real reductive groups, standard Whittaker $(\mathfrak{g},K)$-modules are defined in terms of functions on $G/N$ satisfying certain equivariance and eigenvalue conditions, with degenerate models arising when the defining character is trivial on subsets of the root subgroups (Taniguchi, 2011).
• The composition series of these modules in integral infinitesimal character cases (e.g., for $U(n,1)$) are determined via explicit K-type shift operators and Gelfand–Tsetlin bases, with the nondegenerate case serving as a "model case" for more symmetric or intricate degeneracies present in degenerate settings.
• In Whittaker categories for Lie superalgebras, degeneracy is controlled by the vanishing of the character $\psi$ on certain simple roots, and the associated category decomposes according to generalized eigenspaces for $\psi$, reflecting the breakdown into "types" of Whittaker vectors (Bagci et al., 2012).
• In the construction of degenerate models for induced representations or finite groups (e.g., $GL_{kn}(\mathbb{F}_q)$ or $GL_4(\mathfrak{o}_2)$), the twisted Jacquet module with respect to a degenerate character $\psi$ of a suitable unipotent radical is a key object and often decomposes into sums of explicit, often irreducible, modules for the corresponding Levi subgroup. For strongly cuspidal representations of $GL_4(\mathfrak{o}_2)$, the degenerate Whittaker space is proven to be multiplicity-free (Parashar et al., 30 Jul 2024) and confirms Prasad's conjecture (Gorodetsky et al., 2017, Parashar et al., 14 Aug 2025).

4. Connections to Nilpotent Orbits, Wave-Front Sets, and Automorphic Theory

Degenerate Whittaker models are naturally attached to nilpotent orbits, paralleling the orbit method philosophy in representation theory. The maximal orbits in the Whittaker support for a given $\pi$ are conjecturally quasi-admissible, and for (quasi-)cuspidal representations, $F$-distinguished (Gourevitch et al., 2018). The analytic data of representation theory (e.g., Fourier coefficients of automorphic forms) often localize on degenerate Whittaker models:

• In the Fourier expansion of automorphic forms, especially for "small" or minimal representations, only degenerate Whittaker vectors (nonzero on low-dimensional subgroups) contribute non-vanishing Fourier coefficients, as for minimal representations of $SL(n)$ and exceptional groups (Gustafsson et al., 2014, Fleig et al., 2013).
• The functional dimension of the degenerate model matches the dimension of the dimension of the corresponding nilpotent orbit.
• For $p$-adic groups and automorphic representations, the dimension of the degenerate model coincides with relevant Fourier coefficients and can be calculated via explicit formulas involving $q$-hypergeometric series (Gorodetsky et al., 2017).

5. Functional-Analytic and Combinatorial Aspects

Degenerate Whittaker models appear in analytic settings, such as Markov diffusion models or combinatorial summations:

• Their explicit realizations often involve combinatorial or special-function expansions, such as path models for Whittaker vectors on root lattices, reflecting solutions to the Toda lattice or $q$-difference equations (Francesco et al., 2014).
• In certain analytic or probabilistic models, the large deviation principle for (potentially degenerate) Whittaker models is governed by rate functions that distinguish between strict and degenerate (coalescing) regions, modifying the underlying statistical or energy cost accordingly (Gao et al., 2020).
• The structure of degenerate models is encoded in generating functions, recurrence relations, and specialized expansions in symmetric or supersymmetric functions (e.g., Laguerre polynomials for holomorphic discrete series (Frahm et al., 2022); Jack superpolynomials for superconformal algebras (Desrosiers et al., 2013)).

6. Applications and Broader Context

Degenerate Whittaker models have diverse and profound applications:

• In the local context, they are central to the classification and decomposition of representation categories, the paper of generalized Jacquet modules, and the branching rules for restriction to subgroups (Brown et al., 2019, Gomez et al., 2015).
• Globally, they parameterize Fourier coefficients of automorphic forms associated with nilpotent orbits, control period integrals relevant to $L$-functions and other invariants of automorphic representations, and structure the local and global theta correspondence and periods in the relative trace formula.
• In geometric representation theory and quantum field theory, degenerate Whittaker categories underlie the construction of automorphic sheaves and geometric Eisenstein series, with deep connections to the quantum Langlands correspondence and the theory of perverse sheaves on moduli stacks (Lysenko, 2012).
• In mathematical physics, particularly supersymmetric gauge theories, degenerate Whittaker vectors provide concrete links between representation-theoretic data and partition functions—most notably, in instantiations of the AGT conjecture, their norm squares correspond to combinatorially intricate instanton partition functions (Desrosiers et al., 2013).

7. Summary Table of Key Structural Features

Context	Construction of Degeneracy	Key Invariant/Decomposition
Real or $p$-adic groups	Degenerate character on $N$	Attached nilpotent orbit/associated variety
Finite groups	Twisted Jacquet module	Induced representation on Levi
Lie superalgebras	Singular $\psi$ on nilpotent	Parabolically induced from smaller Levi
Automorphic forms	Partial Fourier integrals	Only orbits $\leq$ wave-front set survive

Degenerate Whittaker models, as a unifying language, link geometric, analytic, categorical, and combinatorial phenomena in representation theory and beyond, allowing the systematic paper of representations beyond the generic/nondegenerate regime and the extraction of finer invariants sensitive to nilpotent orbit data. Their versatility impacts branching problems, harmonic analysis, special function theory, noncommutative geometry, and quantum field theory.