Concept-Level Orthogonality Diagnostics

• Concept-Level Orthogonality Diagnostics are post-hoc tools that decompose model signal energy into interpretable, orthogonal components for deeper complexity analysis.
• They utilize explicit orthogonal basis constructions (via techniques like Gram–Schmidt or orthogonal polynomial kernels) and indices such as ORCA and OKC to attribute model capacity.
• These diagnostics reveal redundancy, interaction effects, and modularity in models, guiding effective model reduction and improved interpretability.

Concept-level orthogonality diagnostics comprise a family of post-hoc analytical tools designed to quantify, attribute, and structurally interpret the allocation of "signal energy"—typically measured as squared norm within a function space—across orthogonal, semantically meaningful components of a learned model. The methodology entails either constructing or enforcing an explicit orthogonal basis in the model's feature space (e.g., via orthogonal polynomial kernels or explicit Gram–Schmidt orthogonalization of decoder dictionaries), followed by development of diagnostic indices that measure how model capacity, statistical dependence, or representational support are distributed across interpretable units such as constant offsets, marginal variables, pairwise or higher-order interactions, and distinct concept features. These diagnostics illuminate sources of complexity, redundancy, and interference not captured by accuracy metrics alone, and underpin recent advances in both kernel methods and neural network interpretability.

1. Orthogonal Basis Construction and Model Expansion

In kernel-based models, particularly SVMs utilizing finite polynomial kernels, an explicit orthonormal basis can be derived by truncating a family of orthogonal polynomials (such as Legendre or Jacobi polynomials) with respect to a chosen Borel measure on the domain. For $d$-dimensional input, a tensor-product construction yields a basis $\{\varphi_j\}_{j=1}^D$ for the RKHS $\mathcal{H}_n^{(d)}$, allowing any $h(x)$ in the space to be written as $h(x) = \sum_{j=1}^D w_j \varphi_j(x)$ with $w_j$ given exactly by kernel evaluations and the support vector coefficients. This approach exposes each basis direction and its contribution to the overall function, laying the foundation for orthogonality-based attribution (Soto-Larrosa et al., 16 Apr 2026).

In neural representation learning, imposing a penalty on the off-diagonal terms of the decoder dictionary's Gram matrix during training leads to feature vectors that are nearly orthogonal. This penalty takes the form

$$\mathcal{L}(x) = \frac{\|x - \hat{x}\|_2^2}{\|x\|_2^2} + \lambda \|\mathrm{tril}(W_D^\top W_D)\|_F^2$$

where $W_D$ is the decoder matrix and $\lambda$ tunes the strength of orthogonality regularization, penalizing intra-feature correlations. As $λ$ increases, dictionary self-coherence $\{\varphi_j\}_{j=1}^D$0 and semantic overlap both decrease, promoting concept separation (Miller et al., 4 Feb 2026).

2. Orthogonal Representation Contribution Analysis (ORCA) and OKC Indices

Once an explicit orthogonal basis is available, diagnostic techniques such as Orthogonal Representation Contribution Analysis (ORCA) can be deployed. Here, the squared RKHS norm of the classifier or function,

$\{\varphi_j\}_{j=1}^D$1

is decomposed into contributions from subsets of basis elements corresponding to structural features, such as interaction order $\{\varphi_j\}_{j=1}^D$2 (number of active coordinates in multi-index $\{\varphi_j\}_{j=1}^D$3) or total polynomial degree $\{\varphi_j\}_{j=1}^D$4. The normalized Orthogonal Kernel Contribution (OKC) index for a subset $\{\varphi_j\}_{j=1}^D$5 of basis functions is

$\{\varphi_j\}_{j=1}^D$6

Block aggregation yields indices for constant (bias), marginal (single variable), pairwise, up to $\{\varphi_j\}_{j=1}^D$7-wise interactions, as well as degree-based groupings: $\{\varphi_j\}_{j=1}^D$8 This decomposition quantifies the allocation of model complexity and enables post-training structural interpretability without retraining or resorting to surrogate models (Soto-Larrosa et al., 16 Apr 2026).

In neural autoencoders, orthogonality diagnostics extend to metrics such as dictionary coherence, average feature explanation distance (using sentence embeddings), and empirically measured intervention efficacy, directly linking orthogonality to interpretability and causal separability (Miller et al., 4 Feb 2026).

3. Metrics, Diagnostic Procedures, and Implementation

The principal diagnostic indices and algorithmic workflow are as follows:

• Design Matrix Construction: For polynomial kernel SVMs, build the design matrix $\{\varphi_j\}_{j=1}^D$9 with $\mathcal{H}_n^{(d)}$0, then compute $\mathcal{H}_n^{(d)}$1.
• Norm and Grouped Indices: Compute total squared norm $\mathcal{H}_n^{(d)}$2. Assign to each index $\mathcal{H}_n^{(d)}$3 its interaction order $\mathcal{H}_n^{(d)}$4 and total degree $\mathcal{H}_n^{(d)}$5. Accumulate $\mathcal{H}_n^{(d)}$6 and normalize for OKC indices.
• Marginal and Pairwise: For each coordinate $\mathcal{H}_n^{(d)}$7, define the marginal OKC as the sum over all $\mathcal{H}_n^{(d)}$8 with support only in $\mathcal{H}_n^{(d)}$9; for $h(x)$0 pairs, sum over all $h(x)$1 supported exactly by $h(x)$2.
• Neural Feature Diagnostics: Orthogonality regularization is quantified by self-coherence, generalized Welch bounds, and by changes in cosine similarity of embedded textual feature explanations. Empirical procedures for concept-level interventions involve manipulating latent coefficients and measuring whether only the targeted concept is affected downstream.

Pseudocode provides an explicit post-training procedure for OKC index calculation, requiring only standard linear algebraic operations (Soto-Larrosa et al., 16 Apr 2026).

4. Applications and Examples

Orthogonality diagnostics provide fine-grained interpretability on synthetic and real datasets:

• Double Spiral (d=2): As polynomial degree $h(x)$3 increases, higher-order and pairwise interaction indices (OKC$h(x)$4) rise, reflecting increased model reliance on variable coupling. Symmetry (even/odd contributions) mirrors data properties and kernel choice.
• Echocardiogram Dataset (d=5): High-order indices (OKC$h(x)$5) dominate for large $h(x)$6, while marginal and low-order interactions vanish, indicating reliance on complex joint effects spanning all five variables.

For sparse autoencoded LLM features, increasing the orthogonality penalty $h(x)$7 decreases both dictionary coherence and textual explanation similarity, confirming emergence of more distinct, isolated feature concepts. Intervention metrics reveal that highly orthogonalized features permit more successful, concept-specific edits with minimal unintended interference (Miller et al., 4 Feb 2026).

5. Conditioning, Identifiability, and Model Reduction

In models such as the Nelson–Siegel–Svensson (NSS) yield curve, orthogonal reparametrization via QR decomposition isolates conditioning and identifiability issues. The discrete design matrix $h(x)$8 is decomposed as $h(x)$9; projections onto $h(x) = \sum_{j=1}^D w_j \varphi_j(x)$0 yield orthogonal parameters $h(x) = \sum_{j=1}^D w_j \varphi_j(x)$1, for which the conditional Fisher information is diagonal. A crucial scalar diagnostic, $h(x) = \sum_{j=1}^D w_j \varphi_j(x)$2 (the final diagonal element of $h(x) = \sum_{j=1}^D w_j \varphi_j(x)$3), quantifies the residual after projecting the fourth basis function onto the preceding three. When $h(x) = \sum_{j=1}^D w_j \varphi_j(x)$4 is small, the fourth basis function is nearly collinear with the others, indicating non-identifiability and justifying model reduction to a simpler (three-mode) structure (Flassig et al., 21 Apr 2026). This scalar test separates true structural identifiability limits from numerical instabilities.

6. Theoretical Insights and Significance

Concept-level orthogonality diagnostics generalize the classical matched filtering and energy decomposition principles, allowing researchers to pinpoint precisely which features and interactions a model exploits. OKC indices, self-coherence, and $h(x) = \sum_{j=1}^D w_j \varphi_j(x)$5 all provide interpretable, quantitative lenses on model expressivity, redundancy, and modularity. In neural autoencoders, enforced orthogonality not only supports interpretability but strengthens causal modularity, aligning with the Independent Causal Mechanisms (ICM) principle by ensuring interventions on one feature minimally disturb others (Miller et al., 4 Feb 2026). In kernel methods, post-training diagnostics using ORCA/OKC provide structure substantially richer than aggregate accuracy, uncovering the dimensional and polynomial order complexity that a model leverages.

7. Limitations and Best Practices

While concept-level orthogonality diagnostics yield powerful attributions, their validity and interpretive value depend critically on the choice of basis and model architecture. Imposing orthogonality may trade off with purely predictive performance and does not guarantee statistical independence beyond second moments. For empirical evaluation:

• In polynomial kernel SVMs, ensure adequate truncation degree and verify the stability of spectral OKC peaks.
• For neural autoencoders, use small TopK reconstructions, regularize only the decoder, and monitor task accuracy and interpretability scores to contraindicate over-regularization.
• Report all hyperparameters, prompts, and evaluation subsets for reproducibility.

These methodologies deliver a general, scalable interpretability layer that can be integrated directly into kernel and neural model workflows, offering precise attribution of complexity, symmetry, and modularity beyond the reach of accuracy-based metrics alone (Soto-Larrosa et al., 16 Apr 2026, Miller et al., 4 Feb 2026, Flassig et al., 21 Apr 2026).