Spectral Utilization Index (SUI) Explained

• SUI is a normalized measure quantifying effective spectrum use in both neural networks and RF applications by combining hard and soft rank metrics.
• It is computed from the eigenspectrum of post-activation covariance, balancing dominant-mode participation with tail capacity to derive an effective dimension.
• SUI informs design decisions for network width, pruning, and low-rank adaptations, and parallels indices in RF spectrum management through normalization.

Spectral Utilization Index (SUI) denotes a normalized measure of how effectively a system uses a spectrum-like resource. In feed-forward networks (FFNs) for LLMs, SUI is defined as a harmonic combination of width-normalized hard and soft spectral ranks computed from the eigenspectrum of post-activation covariance, and it is intended to be high only when dominant-mode participation and tail coverage are simultaneously healthy (Jha et al., 1 Oct 2025). In the RF-spectrum literature represented here, the term itself does not appear in MUSE, but compatible utilization indices can be constructed by normalizing utilized, available, or receiver-constrained spectrum-space over discretized space, time, and frequency (Khambekar et al., 2015).

1. Latent-space utilization in feed-forward networks

The FFN usage of SUI begins from a spectral utilization problem: the relevant question is not only how wide an FFN is, but how much of the added latent capacity is effectively used. Rather than auditing parameter count or FLOPs, the method audits how variance is distributed across FFN latent directions. Each FFN produces a $D$-dimensional post-activation latent vector per token. Over a batch of tokens, the $D \times D$ covariance $\Sigma$ of these activations and its eigenspectrum $\lambda_1 \ge \lambda_2 \ge \dots \ge \lambda_D$ describe how variance is allocated across directions. Directions with large eigenvalues correspond to dominant modes, while many small eigenvalues form a low-energy tail. A direction is “meaningfully activated” if it carries non-negligible variance in this eigenspectrum (Jha et al., 1 Oct 2025).

Within this framework, Hard Rank, Soft Rank, and Spectral Concentration play complementary roles. Hard Rank is a participation-ratio statistic and is described as a spike-sensitive count of dominant directions. Soft Rank is an entropic rank and tracks how broadly variance is spread, responding to tail growth. Spectral Concentration measures how strongly variance is front-loaded into early principal components. SUI combines the normalized hard and soft ranks so that either collapse into a narrow dominant subspace or excessive dilution into a weak tail lowers the index.

The same paper also defines an effective dimension, eDim, as a re-expression of SUI on the original width scale. This converts a unit-interval utilization score back into an “active components” count between $1$ and $D$, making the metric more directly interpretable when comparing width sweeps.

2. Formal definition and derived quantities

Let $X \in \mathbb{R}^{N \times D}$ be the post-activation matrix at an FFN layer for $N$ tokens, let $d$ be the model embedding dimension, and let $D$ be the FFN hidden width. The unbiased covariance and eigendecomposition are

$D \times D$0

$D \times D$1

where $D \times D$2 is the row-wise mean of $D \times D$3. All spectral metrics depend only on the sorted eigenvalues (Jha et al., 1 Oct 2025).

Hard Rank uses the participation ratio on the raw eigenvalues:

$D \times D$4

It is normalized to $D \times D$5 by

$D \times D$6

Soft Rank first normalizes eigenvalues to probabilities,

$D \times D$7

then defines entropy with the natural logarithm,

$D \times D$8

and sets

$D \times D$9

$\Sigma$0

Spectral Concentration uses the cumulative variance curve

$\Sigma$1

and the uniform baseline

$\Sigma$2

to form

$\Sigma$3

Its range is $\Sigma$4; higher values indicate that variance is concentrated in early components, while lower values indicate a more uniform spread.

The Spectral Utilization Index harmonically combines the two normalized ranks:

$\Sigma$5

The index lies in $\Sigma$6 and is high only when both dominant-mode participation and tail coverage are healthy. Effective dimension is then defined as

$\Sigma$7

This mapping translates SUI into an intuitive count of active components between $\Sigma$8 and $\Sigma$9.

3. Computation in decoder-only Transformers

The computation pipeline is defined on post-activation FFN outputs in decoder-only Transformers. For gated FFNs such as SwiGLU, the paper writes $\lambda_1 \ge \lambda_2 \ge \dots \ge \lambda_D$0 and $\lambda_1 \ge \lambda_2 \ge \dots \ge \lambda_D$1, where $\lambda_1 \ge \lambda_2 \ge \dots \ge \lambda_D$2 is SiLU and $\lambda_1 \ge \lambda_2 \ge \dots \ge \lambda_D$3 is elementwise product. The spectral audit is performed on $\lambda_1 \ge \lambda_2 \ge \dots \ge \lambda_D$4, and spectra are computed from the eigenvalues of the covariance $\lambda_1 \ge \lambda_2 \ge \dots \ge \lambda_D$5, not from the SVD of $\lambda_1 \ge \lambda_2 \ge \dots \ge \lambda_D$6 directly (Jha et al., 1 Oct 2025).

For a given layer and training step, the procedure is sequential. One first gathers the $\lambda_1 \ge \lambda_2 \ge \dots \ge \lambda_D$7 post-activation matrix across tokens, computes the row-wise mean, and centers the activations. One then forms the covariance, computes and sorts the eigenvalues, and uses those eigenvalues to compute Hard Rank, Soft Rank, Spectral Concentration, SUI, and eDim. If needed, a tiny $\lambda_1 \ge \lambda_2 \ge \dots \ge \lambda_D$8 may be added only where required for numerical stability, for example when forming $\lambda_1 \ge \lambda_2 \ge \dots \ge \lambda_D$9 if $1$0 must be guaranteed.

Aggregation is performed across layers and training steps, and the paper often reports layer medians together with across-layer variability. The training setup uses C4 data and width sweeps of the form $1$1 with $1$2. Once the eigenvalues are known, the computation of ranks, concentration, and SUI is $1$3; covariance eigendecomposition is the dominant cost.

The practical defaults stated in the paper are to use post-activation covariance, natural logarithm for entropy, float32 or float64 arithmetic, and a small $1$4 only if needed to avoid $1$5. When the metrics are tracked over training, stabilization is reported after approximately $1$6K steps.

4. Spectral scaling laws and architectural effects

The central empirical result is an asymmetric spectral scaling law: across LLaMA backbones, soft rank scales nearly linearly with FFN width, while hard rank grows sublinearly and with substantially higher variance. The paper summarizes the log-log fits as follows (Jha et al., 1 Oct 2025):

Backbone	Soft rank	Hard rank
LLaMA-70M (Pre-LN)	$1$7, $1$8	$1$9, $D$0
LLaMA-130M (Pre-LN)	$D$1, $D$2	$D$3, $D$4
LLaMA-250M (Pre-LN)	$D$5, $D$6	$D$7, $D$8

In the LLaMA-130M example highlighted in the introduction, the paper reports soft rank $D$9 with $X \in \mathbb{R}^{N \times D}$0 versus hard rank $X \in \mathbb{R}^{N \times D}$1 with $X \in \mathbb{R}^{N \times D}$2. The stated functional forms are $X \in \mathbb{R}^{N \times D}$3 with $X \in \mathbb{R}^{N \times D}$4–$X \in \mathbb{R}^{N \times D}$5, and $X \in \mathbb{R}^{N \times D}$6 with $X \in \mathbb{R}^{N \times D}$7–$X \in \mathbb{R}^{N \times D}$8. Hard rank exhibits higher variance, occasional crossovers, and sensitivity to training transients.

After training stabilizes at roughly $X \in \mathbb{R}^{N \times D}$9K steps, the width-normalized utilization exponents in LLaMA-130M Pre-LN converge to approximately $N$0 and $N$1. The paper states this as confirmation that normalized hard rank tends to decrease with width while normalized soft rank is near-flat or slightly increasing: $N$2 and $N$3.

Spectral Concentration is calibrated in the paper using synthetic spectra of the form $N$4. Under that construction, $N$5 increases monotonically with $N$6 from approximately $N$7 at $N$8 to approximately $N$9 at $d$0. For $d$1, at least $d$2 variance resides in the top approximately $d$3 principal components and is described as collapse; for $d$4, at least $d$5 of dimensions are required to explain approximately $d$6 variance and this is described as dilution. Empirically, typical LLaMA-like activations exhibit $d$7–$d$8.

The representative eDim values reported in the paper are sublinear and remain a small fraction of $d$9 across widths. For LLaMA-70M, illustrative layer-median eDim values are approximately $D$0 $D$1, $D$2 $D$3, $D$4 $D$5, $D$6 $D$7, $D$8 $D$9, $D \times D$00 $D \times D$01, $D \times D$02 $D \times D$03, and $D \times D$04 $D \times D$05. For LLaMA-130M the corresponding values are approximately $D \times D$06, $D \times D$07, $D \times D$08, $D \times D$09, $D \times D$10, $D \times D$11, $D \times D$12, and $D \times D$13. For LLaMA-250M they are approximately $D \times D$14, $D \times D$15, $D \times D$16, $D \times D$17, $D \times D$18, $D \times D$19, $D \times D$20, and $D \times D$21. These numbers are used in the paper to argue that widening predominantly expands tail capacity and that gains can be non-monotonic across layers.

The same study reports substantial dependence on normalization strategy and model family. Pre-LN shows the “classic” asymmetry. Post-LN suppresses tail growth, with soft-rank exponents approximately $D \times D$22–$D \times D$23, slightly improves hard-rank exponents to approximately $D \times D$24–$D \times D$25, but is prone to collapse in LLaMA-250M at larger widths, where perplexity spikes above $D \times D$26. Mix-LN restores near-linear soft scaling, approximately $D \times D$27–$D \times D$28, and boosts hard scaling to approximately $D \times D$29–$D \times D$30. Adding WeightNorm or hyperspherical normalization in Post-LN LLaMA-250M prevents collapse and improves perplexity; the paper gives the example of $D \times D$31 perplexity near $D \times D$32 with WeightNorm versus approximately $D \times D$33 for vanilla Post-LN. In the GPT-2 versus nGPT comparison, vanilla GPT-2 shows strong asymmetry, while nGPT narrows the soft-hard gap, reduces concentration to approximately $D \times D$34 versus GPT-2’s approximately $D \times D$35, and improves performance; the stated $D \times D$36 perplexities are $D \times D$37 for nGPT versus approximately $D \times D$38–$D \times D$39 for GPT-2 variants.

5. Design guidance, robustness, and limitations

The stated interpretation is that FFN widening mainly adds low-energy tail directions while dominant-mode subspaces saturate early. SUI is designed to penalize both failure modes: if the core collapses, $D \times D$40 is low; if variance is spread too diffusely, $D \times D$41 is low. The paper therefore recommends monitoring SUI and eDim during width sweeps and treats width selection as a trade-off between tail capacity and dominant-mode capacity (Jha et al., 1 Oct 2025).

For inference-efficient design, the paper reports that gains in eDim per unit $D \times D$42 drop beyond approximately $D \times D$43–$D \times D$44 for LLaMA-like setups under standard training. It gives the heuristic to prefer widths where SUI increases steadily and eDim grows faster than linear in compute cost, and to reallocate budget to depth or other components once SUI plateaus. If SUI plateaus and eDim$D \times D$45 stagnates, extra width is described as tail-only. In regimes where hard rank stops growing and Spectral Concentration remains high, the paper states that narrowing FFNs or adding normalization such as Mix-LN, WeightNorm, or hyperspherical normalization can improve SUI more than raw widening.

The article’s operational implications extend to pruning, low-rank adaptation, and Mixture-of-Experts. Layers with persistently low SUI or eDim at large $D \times D$46 are identified as candidates for width reduction or structured pruning. If $D \times D$47 is low and $D \times D$48 is high, the dominant subspace is small, and low-rank approximations or adapters can target those active directions. Since widening primarily adds tail capacity, Mixture-of-Experts is presented as a mechanism that can allocate experts to tail directions more efficiently than uniformly widening dense FFNs.

Several caveats are explicit. The results are reported for English decoder-only models trained on C4, so spectral use may shift with domain or multilingual corpora. Spectral behavior varies by depth, and the paper reports layer medians and variability rather than claiming uniform layerwise behavior. Early training stages show crossovers and higher variance, especially for hard rank, and scaling exponents stabilize only after roughly $D \times D$49K steps. The work focuses on activation covariance spectra rather than weight spectra, because weight spectra do not directly measure representational usage under the data distribution. It also states that causality between spectral metrics and perplexity is suggestive but not proven, and that the study spans $D \times D$50M–$D \times D$51M parameter backbones, so behavior at much larger scales may differ.

The proposed future directions are extending the analysis to encoder-decoder and multilingual settings, combining SUI with task-specific probes, exploring differentiated width allocation across layers, and developing closed-form or online estimators for training-time control. The paper presents $D \times D$52 as a lightweight estimator for this purpose.

6. RF spectrum-space indices derived from MUSE

A distinct usage arises in spectrum management. In MUSE, spectrum use is characterized in discretized space, time, and frequency, and the total spectrum-space over a region is

$D \times D$53

with

$D \times D$54

At a point $D \times D$55, aggregate occupancy is

$D \times D$56

the unit-spectrum-space occupancy is

$D \times D$57

the unit-spectrum-space opportunity is

$D \times D$58

and the unit-spectrum-space liability is

$D \times D$59

Aggregated over a region, MUSE defines utilized, forbidden, and available spectrum-spaces as $D \times D$60, $D \times D$61, and $D \times D$62, with the conservation law

$D \times D$63

The paper explicitly states that the term Spectral Utilization Index does not appear in MUSE; the following normalized indices are derived to be consistent with its formalism (Khambekar et al., 2015).

The transmitter-level index is the fraction of total spectrum-space occupied by a transmitter:

$D \times D$64

The receiver-level construction has two complementary forms. The Receiver Liability Index is

$D \times D$65

while the Receiver Allowable-Opportunity Index is based on the summed positive interference opportunity and is normalized by $D \times D$66. The system-level form is

$D \times D$67

Related normalized fractions are $D \times D$68 and $D \times D$69.

The worked example reported with the MUSE framework uses a $D \times D$70 km $D \times D$71 $D \times D$72 km region, a hexagonal grid with $D \times D$73 unit regions of side $D \times D$74 m, a unit frequency band of $D \times D$75 MHz, and a unit time-quanta of $D \times D$76 s, giving $D \times D$77, $D \times D$78, and $D \times D$79. With $D \times D$80 W and $D \times D$81 dBm, $D \times D$82 W per unit and $D \times D$83 W·m$D \times D$84. For one transmitter and one receiver, the quantified results are $D \times D$85 W·m$D \times D$86, $D \times D$87 W·m$D \times D$88, and $D \times D$89 W·m$D \times D$90. The derived indices are therefore $D \times D$91, $D \times D$92, $D \times D$93, and $D \times D$94.

The RF formulation is power-centric and differs fundamentally from the FFN formulation. MUSE quantifies RF power occupancy and receiver-imposed constraints in discretized spectrum-space, whereas the FFN version quantifies activation-covariance eigenspectra. The RF indices are contrasted in the source material with spectral efficiency in bits/s/Hz and with channel-occupancy percentages: the former measures information rate, the latter is typically not spatially resolved, while MUSE-style normalized indices separate transmitter occupancy, receiver liability, and remaining opportunity over space, time, and frequency.