Approximately Orthogonal Fine-Tuning (AOFT)

• AOFT is a family of parameter-efficient fine-tuning methods that impose exact or approximate orthogonality on adaptation operators to preserve the geometric structure of pretrained models.
• It unifies diverse approaches—including butterfly factorization, low-rank additive updates, and projection-based techniques—to reduce parameters and maintain performance.
• AOFT methods enhance model generalization and stability across tasks by preserving key spectral and angular properties, improving continual learning and vision adaptations.

Searching arXiv for papers on AOFT and orthogonal fine-tuning. Searching arXiv for "Approximately Orthogonal Fine-Tuning" and related orthogonal PEFT methods. Approximately Orthogonal Fine-Tuning (AOFT) denotes a family of parameter-efficient fine-tuning strategies that impose exact or approximate orthogonality on adaptation operators, low-rank factors, or gradient directions in order to preserve geometric structure during adaptation. In the literature surveyed here, the label is used in more than one sense: as a broad orthogonality-preserving PEFT paradigm rooted in Orthogonal Fine-Tuning (OFT), as a practical butterfly-factorized instantiation of OFT, as a low-rank additive method with orthonormal Stiefel factors in "OrthoGeoLoRA," and as related projection-, support-selection-, and continual-learning schemes (Liu et al., 2023, Wang et al., 14 Jan 2026, Zhao et al., 12 May 2026). This usage pattern suggests that AOFT functions less as a single canonical algorithm than as a geometric design principle for adapting frozen pretrained models.

1. Terminological scope and lineage

A central lineage begins with OFT, in which a frozen pretrained linear layer $W^0\in\mathbb{R}^{d\times n}$ is adapted by an orthogonal matrix $R$, yielding $z=(R\,W^0)^\top x$ with $R^\top R=I_d$. In this formulation, orthogonality is typically enforced by a Cayley parameterization $R=(I+Q)(I-Q)^{-1}$ with $Q^\top=-Q$. Because a dense $d\times d$ orthogonal matrix has $\Theta(d^2)$ free parameters, later work sought more parameter-efficient approximations and structured parameterizations (Liu et al., 2023).

Within that trajectory, one usage of AOFT identifies it with a butterfly-factorized orthogonal update. "Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization" describes AOFT as a practical, parameter-efficient instantiation of OFT in which a dense orthogonal $R$ is replaced by a product of sparse butterfly factors, reducing the footprint from $O(d^2)$ to $R$0 or, more generally, $R$1 (Liu et al., 2023). Related structured formulations include quasi-Givens Orthogonal Fine-Tuning (qGOFT), which uses $R$2 Givens rotations and a soft orthogonality regularizer, and Group-and-Shuffle orthogonal parametrization (GSOFT), which alternates block-diagonal orthogonal transforms with fixed permutations (Ma et al., 2024, Gorbunov et al., 2024).

A second usage of AOFT is additive rather than multiplicative. "OrthoGeoLoRA" defines AOFT as a geometric low-rank replacement for vanilla LoRA, replacing $R$3 by $R$4 with $R$5 and $R$6 constrained to have orthonormal columns on the Stiefel manifold and $R$7 diagonal (Wang et al., 14 Jan 2026). A third usage appears in vision adaptation, where AOFT denotes generation of approximately orthogonal down/up-projection matrices from a single learnable vector so that adapter geometry more closely matches pretrained ViT weights (Yang et al., 17 Jul 2025).

Later work broadens the umbrella still further. LOFT presents a unified multiplicative-subspace-rotation view that recovers coordinate-, butterfly-, Householder-, and principal-subspace-based orthogonal PEFT methods as special cases, while OPLoRA, DOC, FoRA, LoCO, OFTv2, and OrthoFuse each realize orthogonality through projection, dynamic gradient cutting, Stiefel constraints, low-rank compositional rotations, matrix-free Cayley-Neumann updates, or training-free manifold fusion, respectively (Zhao et al., 12 May 2026, Xiong et al., 14 Oct 2025, Zhang et al., 28 Sep 2025, Park et al., 28 May 2026, Nguyen et al., 15 May 2026, Qiu et al., 24 Jun 2025, Aliev et al., 6 Apr 2026).

2. Core mathematical formulations

Despite the diversity of implementations, the surveyed methods fall into a small number of recurrent mathematical templates.

Formulation	Representative update	Orthogonality locus
Multiplicative orthogonal adaptation	$R$8 or $R$9	$z=(R\,W^0)^\top x$0
Additive orthogonalized low-rank adaptation	$z=(R\,W^0)^\top x$1	$z=(R\,W^0)^\top x$2
Orthogonal-complement projection	$z=(R\,W^0)^\top x$3	update restricted away from top-$z=(R\,W^0)^\top x$4 singular subspaces
Subspace rotation	$z=(R\,W^0)^\top x$5	$z=(R\,W^0)^\top x$6, $z=(R\,W^0)^\top x$7

The multiplicative form is the canonical OFT view. Here the adapted weight is obtained by left- or right-multiplying a frozen weight by an orthogonal matrix, preserving angular structure exactly when orthogonality is exact. BOFT, GSOFT, OFTv2, qGOFT, and LoCO all belong to this broad class, though they differ in whether $z=(R\,W^0)^\top x$8 is dense, block-structured, butterfly-factorized, approximated by a truncated Neumann series, or composed from low-rank skew-symmetric generators (Liu et al., 2023, Ma et al., 2024, Gorbunov et al., 2024, Qiu et al., 24 Jun 2025, Nguyen et al., 15 May 2026).

The additive low-rank form preserves the LoRA-style mergeability of $z=(R\,W^0)^\top x$9 but modifies its geometry. In OrthoGeoLoRA, $R^\top R=I_d$0 is explicitly SVD-like: $R^\top R=I_d$1, $R^\top R=I_d$2, and $R^\top R=I_d$3. The forward map uses unconstrained pre-parameters $R^\top R=I_d$4 and constructs $R^\top R=I_d$5, $R^\top R=I_d$6, and $R^\top R=I_d$7 on each forward pass, while remaining compatible with AdamW and standard fine-tuning pipelines (Wang et al., 14 Jan 2026).

Projection-based methods impose orthogonality relative to a chosen subspace rather than on the adapter factors themselves. OPLoRA computes the SVD $R^\top R=I_d$8, forms $R^\top R=I_d$9 and $R=(I+Q)(I-Q)^{-1}$0, and inserts them into the LoRA update so that $R=(I+Q)(I-Q)^{-1}$1. In this construction, the top-$R=(I+Q)(I-Q)^{-1}$2 singular triples of $R=(I+Q)(I-Q)^{-1}$3 remain exactly preserved (Xiong et al., 14 Oct 2025).

LOFT generalizes multiplicative orthogonal adaptation as a subspace rotation. With a row-orthonormal basis $R=(I+Q)(I-Q)^{-1}$4 and an in-subspace orthogonal transform $R=(I+Q)(I-Q)^{-1}$5, it defines

$R=(I+Q)(I-Q)^{-1}$6

so that

$R=(I+Q)(I-Q)^{-1}$7

This explicitly separates the support $R=(I+Q)(I-Q)^{-1}$8 from the in-subspace transform $R=(I+Q)(I-Q)^{-1}$9 (Zhao et al., 12 May 2026).

3. Geometric motivations and theoretical claims

The principal motivation of AOFT is preservation of pretrained geometry. In the OFT formulation $Q^\top=-Q$0 with $Q^\top=-Q$1, cosine similarities between weight vectors remain unchanged. BOFT describes the resulting benefits as angle preservation, spectral stability, and geometric regularity, while Group-and-Shuffle methods treat orthogonal adapters as structured Lie-submanifold elements (Liu et al., 2023, Gorbunov et al., 2024, Aliev et al., 6 Apr 2026).

For additive low-rank adaptation, OrthoGeoLoRA identifies three geometric pathologies of standard LoRA. First, gauge freedom: for any invertible $Q^\top=-Q$2, $Q^\top=-Q$3 leaves $Q^\top=-Q$4 unchanged. Second, scale ambiguity: $Q^\top=-Q$5 for any $Q^\top=-Q$6. Third, rank collapse: unconstrained gradients can make the columns of $Q^\top=-Q$7 and $Q^\top=-Q$8 collinear, so that only a few singular modes survive. The AOFT remedy is the SVD blueprint: orthonormal bases $Q^\top=-Q$9, no gauge or scale ambiguity, and diagonal $d\times d$0 with explicit magnitudes (Wang et al., 14 Jan 2026).

Other theoretical arguments center on preserving important subspaces. OPLoRA proves that if $d\times d$1, then for each $d\times d$2,

$d\times d$3

so the top-$d\times d$4 singular triples of the frozen weight remain exact singular triples after fine-tuning. To quantify interference, it introduces

$d\times d$5

where low $d\times d$6 indicates low forgetting risk (Xiong et al., 14 Oct 2025).

In continual learning, DOC advances a related but dynamic view. It attributes long-term forgetting to drift of functional directions during fine-tuning and projects new LoRA gradients to be orthogonal to a tracked low-dimensional basis of historical functional directions obtained by Online PCA. The exact orthogonal projection $d\times d$7 is replaced by a low-rank, dynamically updated approximation (Zhang et al., 28 Sep 2025).

The ViT-specific AOFT literature ties orthogonality to generalization. It reports that pretrained ViT weight matrices exhibit approximate orthogonality among row and column vectors, whereas independently learned down/up-projection matrices do not. Under this account, approximate orthogonality reduces the operator norm $d\times d$8, tightening a Rademacher-complexity-based upper bound on generalization error. The argument uses $d\times d$9-approximately orthogonal columns $\Theta(d^2)$0 satisfying $\Theta(d^2)$1 and obtains $\Theta(d^2)$2 by Geršgorin (Yang et al., 17 Jul 2025).

4. Parameterizations, optimization, and computational structure

A defining feature of AOFT research is the search for orthogonality-preserving parameterizations that are compatible with standard optimizers and practical budgets.

Method	Parameterization or optimizer	Stated scaling or budget
BOFT	product of sparse butterfly orthogonal factors	$\Theta(d^2)$3 or roughly $\Theta(d^2)$4 parameters
qGOFT	$\Theta(d^2)$5 Givens rotations or $\Theta(d^2)$6 soft blocks	$\Theta(d^2)$7 parameters
OrthoGeoLoRA	$\Theta(d^2)$8 with differentiable orthonormalization	$\Theta(d^2)$9 trainable parameters
OFTv2	input-centric matrix-free Cayley-Neumann reformulation	per-layer cost $R$0

BOFT factorizes a dense orthogonal map into butterfly stages. For $R$1, each stage uses a block-diagonal butterfly component, and the product $R$2 yields a practical approximation with strong expressivity. The paper states that BOFT is strictly more expressive than block-diagonal OFT of the same block size and can reach the entire orthogonal group through suitably many butterfly factors (Liu et al., 2023).

qGOFT reduces quadratic parameterization cost by using Givens rotations. It notes that any $R$3 can be factorized into at most $R$4 plane rotations, while its practical parameterization uses

$R$5

together with a quasi-orthogonal relaxation in which each $R$6 rotation block is replaced by a learnable matrix $R$7 and regularized by $R$8. This permits flexible norm and relative angular adjustments under soft orthogonality regularization (Ma et al., 2024).

OrthoGeoLoRA keeps the low-rank additive LoRA workflow while changing its internal geometry. Its forward cost matches LoRA at $R$9, with an extra $O(d^2)$0 for the diagonal multiply, and its training-time orthogonalization overhead is $O(d^2)$1 per update. The paper states that in practice, for $O(d^2)$2 and $O(d^2)$3, this overhead is negligible compared to the base layer’s $O(d^2)$4, and that $O(d^2)$5 can be folded into $O(d^2)$6 at inference (Wang et al., 14 Jan 2026).

OFTv2 shifts orthogonal fine-tuning from a weight-centric to an input-centric implementation. Instead of explicitly forming $O(d^2)$7, it computes $O(d^2)$8 and then $O(d^2)$9. It further replaces the exact inverse in the Cayley transform by a truncated Neumann series, producing the Cayley-Neumann parameterization. The stated consequence is a per-layer complexity reduction from $R$00 to $R$01, together with up to $R$02 faster training and $R$03 lower GPU memory usage (Qiu et al., 24 Jun 2025).

Additional parameterizations expand the design space. LoCO builds rotations from low-rank skew-symmetric matrices $R$04, maps them to $R$05, and approximates a product of rotations by a first-order sum $R$06, with orthogonality deviation bounded by $R$07 when $R$08 (Nguyen et al., 15 May 2026). FoRA enforces column orthonormality of the LoRA down-projection $R$09 via Riemannian gradient projection and QR- or Cayley-based retraction, and selects adapted layers by a single-pass diagonal Fisher score (Park et al., 28 May 2026).

5. Empirical performance across tasks and modalities

The empirical record reported in these papers spans retrieval, language understanding, reasoning, vision transfer, diffusion generation, continual learning, and quantized adaptation.

Setting	Representative result	Paper
ELSST hierarchical concept retrieval, multilingual-e5-small, $R$10	OrthoGeoLoRA: MRR $R$11, R@3 $R$12; LoRA: MRR $R$13, R@3 $R$14	(Wang et al., 14 Jan 2026)
GLUE with DeBERTaV3-base	qGOFT average $R$15; GOFT $R$16; LoRA $R$17	(Ma et al., 2024)
RoBERTa-base on GLUE	GSOFT $R$18; LoRA $R$19; OFT $R$20	(Gorbunov et al., 2024)
LLaMA-family commonsense suite	FoRA outperforms LoRA/DoRA by $R$21–$R$22 points at roughly half params	(Park et al., 28 May 2026)
LLaMA-2 7B forgetting score	OPLoRA-128 $R$23; LoRA $R$24	(Xiong et al., 14 Oct 2025)
LLaMA-7B continual learning	DOC: AA $R$25 vs $R$26, BWT $R$27 vs $R$28 relative to O-LoRA	(Zhang et al., 28 Sep 2025)
Quantized LLaMA-2-7B on GSM8K	QOFT-NF4 $R$29; QLoRA-NF4 $R$30	(Qiu et al., 24 Jun 2025)

Within additive low-rank adaptation, OrthoGeoLoRA reports that AOFT outperforms standard LoRA and several PEFT variants under the same low-rank budget on ELSST concept retrieval. The singular-value analysis shows a steep decay for LoRA’s $R$31 but a flat learned $R$32 for AOFT, which the paper interprets as full use of rank $R$33; rank ablation further shows AOFT outperforming LoRA at every tested rank, with AOFT($R$34) exceeding LoRA($R$35) (Wang et al., 14 Jan 2026).

In orthogonal multiplicative PEFT, BOFT reports competitive or superior results across DeBERTaV3-base on GLUE, LLaMA-2-7B on MMLU and GSM8K/MATH, DINOv2-large on VTAB-1k, SAM on HQSeg-44K, and Stable Diffusion with ControlNet or DreamBooth, while using fewer parameters than dense OFT (Liu et al., 2023). LoCO similarly reports $R$36 mean GLUE score on DeBERTa-V3-base, $R$37 on GSM8K/MATH for LLaMA2-7B, and $R$38 on VTAB-1k with ViT-B/16 and $R$39M trainable parameters (Nguyen et al., 15 May 2026).

In vision-specific approximate orthogonality, the ViT AOFT paper reports for ViT-B/16 on FGVC that Adapter+AOFT improves from $R$40 to $R$41 while reducing parameters from $R$42M to $R$43M, and LoRA+AOFT improves from $R$44 to $R$45 with $R$46M parameters. On VTAB-1k, it reports LoRA improving from $R$47 to $R$48 with AOFT and parameter count $R$49M (Yang et al., 17 Jul 2025).

Continual- and forgetting-oriented orthogonal methods report a different empirical emphasis. OPLoRA states that standard LoRA and PiSSA show $R$50–0.6 across layers, while OPLoRA-128 drives $R$51 to near zero for all $R$52, and also improves Python code generation on MBPP/MBPP++ to pass@1 $R$53 relative to LoRA’s $R$54 (Xiong et al., 14 Oct 2025). DOC reports consistent gains on standard and long-chain continual learning benchmarks for LLaMA-7B, LLaMA-13B, and T5-Large, with an ablation showing that freezing the PCA basis drops performance back to O-LoRA levels (Zhang et al., 28 Sep 2025).

6. Conceptual distinctions, misconceptions, and open directions

A recurrent misconception is that AOFT names one method. The surveyed literature contradicts this directly: the term is used for butterfly-factorized orthogonal OFT, for orthonormalized low-rank LoRA updates, for approximately orthogonal adapter factors in ViTs, and more generally for orthogonality-preserving PEFT strategies that differ in whether they are multiplicative, additive, projected, or dynamic (Liu et al., 2023, Wang et al., 14 Jan 2026, Yang et al., 17 Jul 2025).

A second misconception is that orthogonality is always enforced exactly. Some methods do maintain exact constraints: Cayley-parameterized OFT blocks, GS orthogonal matrices, Stiefel-constrained OrthoGeoLoRA factors, and FoRA’s down-projection all preserve orthogonality by construction or retraction. Others are explicitly approximate: qGOFT uses a soft orthogonality penalty; OFTv2 truncates the Neumann series; LoCO uses a first-order approximation to a compositional rotation chain; the ViT AO operator drops the unit-norm constraint on $R$55, leaving columns “approximately orthogonal” (Ma et al., 2024, Qiu et al., 24 Jun 2025, Nguyen et al., 15 May 2026, Yang et al., 17 Jul 2025).

A third misconception is that orthogonal parameterization alone is the decisive design choice. LOFT argues that orthogonal adaptation conflates two distinct components—support selection and in-subspace transform—and presents a first-order analysis in which useful supports should be informed by the downstream training signal. Its gradient-informed supports, SkewGrad and GradSVD, improve the efficiency-performance frontier under matched parameter, memory, and compute budgets, indicating that the choice of subspace $R$56 can be as important as the orthogonal transform $R$57 (Zhao et al., 12 May 2026).

The current frontier extends beyond single-task adaptation. OrthoFuse shows that structured orthogonal adapters can be fused without further training by approximate geodesic interpolation on the Group-and-Shuffle manifold followed by a spectra-restoration transform, reporting a user-study overall preference of $R$58 vs $R$59 against K-LoRA and $R$60 vs $R$61 against ZipLoRA in subject-style fusion (Aliev et al., 6 Apr 2026). OFTv2 extends orthogonal fine-tuning to quantized models; DOC adds dynamic orthogonalization for continual learning; FoRA combines Stiefel constraints with Fisher-based layer selection; and the ViT AOFT work explicitly proposes extension to other architectures and modalities (Qiu et al., 24 Jun 2025, Zhang et al., 28 Sep 2025, Park et al., 28 May 2026, Yang et al., 17 Jul 2025).

Taken together, these developments indicate that AOFT has evolved into a geometric research program organized around several recurring questions: which subspace should be adapted, how orthogonality should be parameterized or approximated, what spectral or functional directions should be preserved, and how these choices interact with parameter budgets, quantization, continual learning, and adapter composition.