Adaptive Orthogonality in Neural Networks

• Adaptive Orthogonality (AO) is a set of principles that enforces dynamic, often approximate, orthogonality in neural networks to improve robustness, generalization, and stability.
• The approach includes techniques such as soft penalties on gradient conflicts, hard projections via polar decomposition, and parameter-level orthogonalizations for models like LLMs and transformers.
• Empirical results demonstrate AO’s benefits in tasks like robust unlearning, parameter-efficient fine-tuning, and adversarial defense, yielding improved accuracy and reduced model redundancy.

Adaptive Orthogonality (AO) refers to a set of principles and mechanisms designed to enforce or optimize orthogonality properties of neural network parameters or gradients, dynamically or approximately, for the purposes of improved robustness, generalization, and stability. AO has emerged as a unifying concept in several subfields, where strict or soft orthogonality constraints are adaptively enforced in training objectives, network parameterizations, or optimization dynamics. The main drivers of AO are the mitigation of destructive interference between conflicting objectives (such as unlearning versus retention), reduction of model redundancy, enhanced robustness to noise or adversarial attacks, and alignment of internal representations for improved generalization.

1. Formal Definitions and Mathematical Foundations

AO can be instantiated at multiple levels: as explicit penalties on parameter orthogonality, as geometric regularizers on loss gradients, or as construction principles for low-rank or convolutional layers.

Loss-Level Adaptive Orthogonality Penalty

In robust machine unlearning, AO penalizes geometric conflicts between gradient directions associated with forget and retain objectives. Denote model parameters as $\theta$, with two data splits:

• $\mathcal{D}_f$: Forget set, loss $L_{\mathrm{forget}}(h_f;\theta)$ (maximized to erase knowledge)
• $\mathcal{D}_r$: Retain set, loss $L_{\mathrm{retain}}(h_r;\theta)$

Let $g_f := \nabla_\theta L_{\mathrm{forget}}$, $g_r := \nabla_\theta L_{\mathrm{retain}}$. The cosine similarity is: $$\cos(g_f, g_r) = \frac{g_f \cdot g_r}{\|g_f\| \|g_r\|}$$ An AO regularizer is applied only when $g_f \cdot g_r < 0$: $$\mathcal{R}_{\rm AO} = \mathbb{I}(g_f \cdot g_r < 0) \left( \frac{1 - \cos(g_f, g_r)}{2} \right)^\gamma$$

$\gamma$ controls curvature and $\lambda_a$ adjusts regularizer strength. The total objective is: $$\mathcal{L}_{\mathrm{unlearn}}(\theta) = L_{\mathrm{forget}}(h_f; \theta) + L_{\mathrm{retain}}(h_r; \theta) + \lambda_a \mathcal{R}_{\rm AO}$$

This penalty enforces a dynamic (adaptive) orthogonality by penalizing conflicting gradient directions (Li et al., 2 Feb 2026).

Parameter-Level Approximate Orthogonality

For parameter-efficient fine-tuning (PEFT) of transformers, AO refers to constructing down/up-projection matrices whose rows/columns are nearly orthogonal: $$|\langle w_i, w_j \rangle| \ll \|w_i\|\|w_j\|, \quad i \neq j$$ Approximately orthogonal matrices have tightly concentrated vector pairwise angles near $90^\circ$ and reduce Rademacher-complexity upper bounds, thereby enhancing generalization (Yang et al., 17 Jul 2025).

2. Algorithmic Instantiations

AO is realized via regularizers (soft), hard constraints, or adaptive relaxation schedules:

• Soft Penalty: Apply spectral norm or Frobenius norm penalty, e.g., $\|X^TX - I\|_F^2$, or the spectral restricted isometry property (SRIP) penalty for DNN layers (Cui et al., 2022).
• Hard Projection: At initialization, polar-decomposition-based orthogonalization projects onto the Stiefel manifold (PDOI), followed by soft adaptation (Cui et al., 2022).
• Parameterization: For PEFT, generate adaptation matrices from a single seed vector using a Householder-style transformation, yielding approximately orthogonal columns/rows (Yang et al., 17 Jul 2025).
• Structural AO: Adaptive Orthogonal Convolutions (AOC) use compositions of block-convolution orthogonal parameterization (BCOP) and reshaped kernel orthogonalization (RKO) to ensure exact orthogonality in convolutional layers under arbitrary kernel, stride, and grouping configurations (Boissin et al., 14 Jan 2025).

3. AO in Robust Unlearning, Generalization, and Robustness

Machine Unlearning: AGT$^{AO}$

In AGT$^{AO}$, AO regularization mediates geometric gradient conflicts between forgetting and retention objectives for LLM unlearning. By adaptively penalizing high-conflict updates, AGT$^{AO}$ achieves near-optimal Knowledge Unlearning Ratio (KUR $\approx$ 0.01), retention-set utility (MMLU 58.30), and robust adversarial defense. Empirical ablations confirm that AO outperforms both no regularization and hard-projection, notably stabilizing cosine-similarity oscillations that otherwise lead to instability or catastrophic forgetting (Li et al., 2 Feb 2026).

PEFT and Fine-Tuning: AOFT

Approximate Orthogonality in AOFT reduces model generalization errors by aligning learned adaptation matrices with the (empirically orthogonal) backbone geometry. AOFT parameterizes adaptation layers so that the spectral and Frobenius norms are minimized, empirically yielding $+3.2\%$ over adapter baselines and $+0.4\%$ for LoRA (with roughly half the adaptation parameters) on fine-grained classification, and a $2\%$ uplift on VTAB-1k (at one-third the parameter count) (Yang et al., 17 Jul 2025).

Orthogonal Convolutions: AOC

AOC layers combine explicit block-convolution orthogonality with reshaping and re-orthogonalization to preserve exact norm or 1-Lipschitz continuity, with full support for all modern convolutional operations. This strict imposition of AO leads to improved adversarial robustness, gradient stability, and scalable computational efficiency. Empirical results demonstrate state-of-the-art certified robust accuracy (CIFAR-10: $60.1\%$ at $\epsilon=36/255$) and near-baseline clean accuracy at modest compute and memory overhead (Boissin et al., 14 Jan 2025).

Robustness to Corruption

TAOTF applies adaptive orthogonality via a two-stage protocol: hard projection (PDOI) followed by global soft penalties, achieving superior performance to conventional training on natural and medical robustness benchmarks without loss in clean accuracy (Cui et al., 2022).

4. Empirical Evaluations

Key empirical findings include:

Setting	AO Method	Main Metric(s)	Result(s)	Reference
LLM unlearning	AGT$^{AO}$	KUR, Model Utility, MMLU	KUR = 0.01, MMLU = 58.30	(Li et al., 2 Feb 2026)
ViT PEFT (FGVC)	LoRA + AOFT / Adapter+AOFT	FGVC mean Acc., Params	89.9% / 88.9% (@0.22M/0.20M)	(Yang et al., 17 Jul 2025)
CNN robustness	TAOTF	CIFAR-100 noisy, APTOS19	+7–10pp gain clean, noise, blur	(Cui et al., 2022)
Orthogonal Conv	AOC	Certified $\ell_2$ accuracy	60.1% (CIFAR-10, $\epsilon$)	(Boissin et al., 14 Jan 2025)

Ablation studies demonstrate AO's superiority over both no regularization (degraded KUR, utility) and hard projection (lower utility, less stability) in unlearning; similar trends are visible in PEFT and convolution applications.

5. Practical Implementation and Guidelines

Instantiating AO requires the selection of tuning hyperparameters and architectural choices:

• AO penalty exponent: $\gamma=1$ is recommended as default; higher values for harsher conflict penalization (Li et al., 2 Feb 2026).
• Regularization schedule: Fixed or adaptive $\lambda_a$; possible annealing as training progresses (Li et al., 2 Feb 2026).
• Projection vs. Penalty: Soft penalty tends to outperform hard-projection in generalization and stability (Li et al., 2 Feb 2026, Cui et al., 2022).
• Layer/parameter selection: In massive models, performance and scalability may require AO to be restricted to a subset of parameters (e.g., only FFN layers, groups) (Li et al., 2 Feb 2026).
• Parameter sharing: For AOFT, each down/up-projection matrix can be generated from a single seed per layer for maximal efficiency; open question remains on expressivity with multiple seeds (Yang et al., 17 Jul 2025).
• Orthogonality maintenance: Re-orthogonalization via QR or Björck iterations at each optimizer step ensures exact AO in convolutional layers (Boissin et al., 14 Jan 2025).

6. Limitations, Extensions, and Open Questions

Known limitations and avenues for extension of AO approaches include:

• Computational overhead: AO penalties and multi-gradient calculations yield $\sim$1.5$\times$ cost per iteration in unlearning; SVD/orthogonalization steps in parameter-space AO can become heavy for large layers, although amortized overhead is low for AOC (Li et al., 2 Feb 2026, Boissin et al., 14 Jan 2025).
• Scalability: Applying AO across all layers/parameters of 70B+ parameter models or large dense layers is time and memory-intensive. Mitigation strategies include layer/parameter group selection or approximate AO (Li et al., 2 Feb 2026).
• Dynamic scheduling: Use of dynamic regularization strength ($\lambda_a$ decay) or per-sample/per-client AO for federated contexts is suggested as future work (Li et al., 2 Feb 2026).
• Alternative conflict or orthogonality metrics: AO is most commonly formulated via cosine similarity, but norm-of-projected-component and other divergence measures are plausible alternatives (Li et al., 2 Feb 2026).
• AOFT expressivity limits: Current AOFT uses a single global seed; multi-seed or combined explicit penalty variants could further strengthen adaptation capacity (Yang et al., 17 Jul 2025).
• Task-specific analysis: Theoretical bounds are necessary but not sufficient to explain empirical gains; NTK or mean-field analyses may clarify the role of AO in generalization (Yang et al., 17 Jul 2025).

7. Significance and Context within the Broader Neural Network Literature

Adaptive Orthogonality, as instantiated across loss geometry, parameterization, initialization, and convolutional layers, offers a consistent mechanism for aligning network structure and optimization to desired robustness, generalization, and stability objectives. AO frameworks have demonstrated marked empirical progress in privacy-preserving LLMs, highly parameter-efficient fine-tuning of large vision models, noise-robust medical imaging pipelines, and adversarially certified classification, all with minimal overhead. AO's ability to mediate fundamental trade-offs—forgetting versus retention, robustness versus accuracy, expressivity versus overfitting—establishes it as a core technique with continuing relevance across varied architectures and application domains (Li et al., 2 Feb 2026, Yang et al., 17 Jul 2025, Cui et al., 2022, Boissin et al., 14 Jan 2025).