NoRA++: Efficient Joint Fine-Tuning

• NoRA++ is a parameter-efficient fine-tuning method that combines activation-space and weight-space adaptation using learnable rational functions.
• It employs structured low-rank perturbations and group-wise parameterization to achieve localized tuning with minimal additional parameters.
• Empirical studies on ViT-Tiny and LLaMA3-8B show that NoRA++ enhances accuracy and generation quality compared to traditional fine-tuning methods.

Searching arXiv for papers that explicitly mention “NoRA++” and related “NoRA” uses to disambiguate the term. Searching arXiv for: "NoRA++ activation functions efficient fine-tuning" NoRA++ is a parameter-efficient fine-tuning configuration in which activation-space tuning and weight-space tuning are trained jointly in a frozen pretrained transformer. In the formulation introduced in "Don't Forget the Nonlinearity: Unlocking Activation Functions in Efficient Fine-Tuning," NoRA replaces fixed activations with learnable rational functions updated through structured low-rank perturbations, while LoRA injects low-rank adapters into selected linear layers; NoRA++ combines these components under a shared training budget to exploit complementary update directions in activation space and weight space (Yin et al., 16 Sep 2025).

1. Terminological scope and disambiguation

Within recent arXiv literature, the label "NoRA" is polysemous. "NoRA: Breaking the Linear Ceiling of Low-Rank Adaptation via Manifold Expansion" introduces NoRA as "Non-linear Rank Adaptation" for PEFT and does not mention "NoRA++"; "When No Paths Lead to Rome" uses "NoRA" as a benchmark for systematic neural relational reasoning and likewise does not define "NoRA++"; "Nora: Normalized Orthogonal Row Alignment for Scalable Matrix Optimizer" uses "Nora" as an optimizer name and also does not mention "NoRA++" (Chen, 26 Feb 2026, Das et al., 27 Oct 2025, Yuan et al., 5 May 2026).

In the PEFT sense relevant here, NoRA++ denotes the composition introduced in (Yin et al., 16 Sep 2025). Its distinctive feature is that it does not restrict adaptation to weight matrices. Instead, it treats activation functions as trainable objects and combines that functional adaptation with standard low-rank weight updates. This places NoRA++ alongside LoRA and DoRA as a PEFT method, but its defining mechanism is the simultaneous use of activation-space and weight-space adaptation.

2. Activation-space formulation

The NoRA component of NoRA++ replaces a fixed activation with a learnable rational function,

$$\phi(x) \;=\; \frac{P(x)}{Q(x)} \;=\; \frac{\sum_{i=0}^{m} a_i x^i}{\sum_{j=0}^{n} b_j x^j}.$$

To avoid numerical instability, training uses the stabilized form

$$\phi(x) \;=\; \frac{a_0 + a_1 x + \cdots + a_m x^m}{1 + \left|\,b_0 + b_1 x + \cdots + b_n x^n\,\right|}.$$

This makes the denominator strictly positive and precludes division by zero, which is particularly important when denominator coefficients are perturbed (Yin et al., 16 Sep 2025).

The numerator and denominator degrees control expressiveness; in ablations for vision, a practical choice is $(m = 5, n = 4)$. For pretrained backbones that originally use GELU in ViT or similar smooth activations in LLaMA MLPs, the base rational activation is chosen to approximate the original fixed nonlinearity. Low-rank update matrices are zero-initialized so that the initial forward pass equals the pretrained behavior. Backpropagation proceeds through the usual activation Jacobian,

$$J_x \;=\; \frac{\partial h}{\partial x} \;=\; \mathrm{Diag}\!\big(\phi'(Wx)\big)\,W.$$

This design reinterprets PEFT as functional modulation rather than only matrix adaptation. The paper states that changing activation shape modulates layer sensitivity, conditioning, and gradient flow without touching the frozen backbone weights. It further states that activation-space tuning opens a new axis of adaptation that complements LoRA’s weight updates (Yin et al., 16 Sep 2025).

3. Structured low-rank and group-wise parameterization

NoRA applies structured low-rank perturbations separately to numerator and denominator coefficients, and it does so group-wise across channels. The hidden dimension of each layer is partitioned into $G$ disjoint groups, and all neurons in a group share one rational activation. This yields $G$ activations per layer rather than per-neuron activations, providing localized flexibility with linear overhead in $G$ (Yin et al., 16 Sep 2025).

For group $g$, the perturbed activation is written as

$$\phi'_g(X) \;=\; \frac{\big(P_g + A^P_g B^P_g\big)(X)}{\big(Q_g + A^Q_g B^Q_g\big)(X)},$$

with

$$\Delta P_g \;=\; A^P_g B^P_g,\qquad \Delta Q_g \;=\; A^Q_g B^Q_g.$$

The low-rank factors satisfy

$$\phi(x) \;=\; \frac{a_0 + a_1 x + \cdots + a_m x^m}{1 + \left|\,b_0 + b_1 x + \cdots + b_n x^n\,\right|}.$$0

where $$\phi(x) \;=\; \frac{a_0 + a_1 x + \cdots + a_m x^m}{1 + \left|\,b_0 + b_1 x + \cdots + b_n x^n\,\right|}.$$1 denotes the polynomial degree for the respective component and $$\phi(x) \;=\; \frac{a_0 + a_1 x + \cdots + a_m x^m}{1 + \left|\,b_0 + b_1 x + \cdots + b_n x^n\,\right|}.$$2 is the chosen rank. This adds only $$\phi(x) \;=\; \frac{a_0 + a_1 x + \cdots + a_m x^m}{1 + \left|\,b_0 + b_1 x + \cdots + b_n x^n\,\right|}.$$3 trainable scalars per component and group. Initialization follows LoRA-style practice: $$\phi(x) \;=\; \frac{a_0 + a_1 x + \cdots + a_m x^m}{1 + \left|\,b_0 + b_1 x + \cdots + b_n x^n\,\right|}.$$4 is initialized with small Gaussian noise, exemplified by $$\phi(x) \;=\; \frac{a_0 + a_1 x + \cdots + a_m x^m}{1 + \left|\,b_0 + b_1 x + \cdots + b_n x^n\,\right|}.$$5, and $$\phi(x) \;=\; \frac{a_0 + a_1 x + \cdots + a_m x^m}{1 + \left|\,b_0 + b_1 x + \cdots + b_n x^n\,\right|}.$$6 is initialized to zeros, guaranteeing that the activation equals the base rational function at the start of fine-tuning (Yin et al., 16 Sep 2025).

The group-wise design is also the mechanism through which locality is controlled. Increasing $$\phi(x) \;=\; \frac{a_0 + a_1 x + \cdots + a_m x^m}{1 + \left|\,b_0 + b_1 x + \cdots + b_n x^n\,\right|}.$$7 increases localized capacity while preserving a compact parameter budget. The paper reports that this structure improves stability and keeps parameter growth sub-linear in the practical scaling study.

4. Composition with LoRA in NoRA++

NoRA++ combines the activation adapter above with LoRA’s weight adapter in the same frozen model. LoRA modifies a selected weight matrix according to

$$\phi(x) \;=\; \frac{a_0 + a_1 x + \cdots + a_m x^m}{1 + \left|\,b_0 + b_1 x + \cdots + b_n x^n\,\right|}.$$8

with $$\phi(x) \;=\; \frac{a_0 + a_1 x + \cdots + a_m x^m}{1 + \left|\,b_0 + b_1 x + \cdots + b_n x^n\,\right|}.$$9 and $(m = 5, n = 4)$0. The forward computation uses $(m = 5, n = 4)$1, while the NoRA component applies $(m = 5, n = 4)$2 in the MLP activation. Training is joint and uses the same task loss as the baseline task, including cross-entropy for classification and supervised fine-tuning loss for instruction tuning (Yin et al., 16 Sep 2025).

In transformer integration, fixed activations in MLP blocks are replaced with group-wise rational activations, while attention components and layer norms remain unchanged. In ViT-Tiny, the GELU in the feed-forward sub-block is replaced. In LLaMA3-8B, the MLP blocks’ activations are replaced with group-wise rational functions. The residual path is preserved.

A standard budget comparison reported for ViT-Tiny is as follows:

Method	Trainable parameters	Share
Full tuning	5.54M	100%
LoRA	0.33M	6.0%
DoRA	0.34M	6.1%
NoRA	0.02M	0.4%
NoRA++	0.35M	6.2%

The paper defines “matched budgets” as using approximately the same number of trainable parameters across methods for fair comparison. In vision experiments, NoRA++ uses approximately $(m = 5, n = 4)$3M trainable parameters, closely matching LoRA and DoRA budgets while remaining far below full fine-tuning (Yin et al., 16 Sep 2025).

5. Empirical performance

On ViT-Tiny pretrained on ImageNet-1K and fine-tuned on CIFAR-10 and CIFAR-100, NoRA alone updates approximately $(m = 5, n = 4)$4M parameters, about $(m = 5, n = 4)$5 of the model, yet reaches $(m = 5, n = 4)$6 on CIFAR-10 and $(m = 5, n = 4)$7 on CIFAR-100, surpassing full fine-tuning by $(m = 5, n = 4)$8 and $(m = 5, n = 4)$9. NoRA++ reaches the best reported accuracies, $$J_x \;=\; \frac{\partial h}{\partial x} \;=\; \mathrm{Diag}\!\big(\phi'(Wx)\big)\,W.$$0 on CIFAR-10 and $$J_x \;=\; \frac{\partial h}{\partial x} \;=\; \mathrm{Diag}\!\big(\phi'(Wx)\big)\,W.$$1 on CIFAR-100, outperforming LoRA and DoRA under matched budgets (Yin et al., 16 Sep 2025).

The same study reports a group-scaling trend: increasing $$J_x \;=\; \frac{\partial h}{\partial x} \;=\; \mathrm{Diag}\!\big(\phi'(Wx)\big)\,W.$$2 from $$J_x \;=\; \frac{\partial h}{\partial x} \;=\; \mathrm{Diag}\!\big(\phi'(Wx)\big)\,W.$$3 to $$J_x \;=\; \frac{\partial h}{\partial x} \;=\; \mathrm{Diag}\!\big(\phi'(Wx)\big)\,W.$$4 with rank $$J_x \;=\; \frac{\partial h}{\partial x} \;=\; \mathrm{Diag}\!\big(\phi'(Wx)\big)\,W.$$5 steadily improves CIFAR-10 accuracy from approximately $$J_x \;=\; \frac{\partial h}{\partial x} \;=\; \mathrm{Diag}\!\big(\phi'(Wx)\big)\,W.$$6 to approximately $$J_x \;=\; \frac{\partial h}{\partial x} \;=\; \mathrm{Diag}\!\big(\phi'(Wx)\big)\,W.$$7, and CIFAR-100 accuracy from approximately $$J_x \;=\; \frac{\partial h}{\partial x} \;=\; \mathrm{Diag}\!\big(\phi'(Wx)\big)\,W.$$8 to approximately $$J_x \;=\; \frac{\partial h}{\partial x} \;=\; \mathrm{Diag}\!\big(\phi'(Wx)\big)\,W.$$9, while trainable parameters rise from approximately $G$0K to approximately $G$1K. Rank ablations show that performance improves up to $G$2 and then saturates. Selective perturbation also matters: perturbing only the numerator or only the denominator degrades CIFAR-10 performance to $G$3 or $G$4, whereas co-perturbation yields $G$5. Alternative learnable activations underperform markedly on CIFAR-100; the paper reports $G$6 for PReLU versus $G$7 for NoRA (Yin et al., 16 Sep 2025).

For LLaMA3-8B instruction tuning, the reported NoRA++ results are given on Alpaca, MathInstruct, OpenOrca, ShareGPT-Hyper, and UltraChat, evaluated on MMLU with 5-shot prompting. The paper states that NoRA++ consistently improves generation quality, with average MMLU gains of $G$8 to $G$9. Reported subset gains include Alpaca STEM $G$0 and Average $G$1, MathInstruct STEM $G$2 and Average $G$3, OpenOrca STEM $G$4, Social Sciences $G$5, and Average $G$6, ShareGPT-Hyper Average $G$7, and UltraChat Average $G$8. The paper also notes that some categories show small regressions, including Humanities $G$9 on OpenOrca (Yin et al., 16 Sep 2025).

6. Theoretical interpretation, efficiency, and limitations

The theoretical interpretation advanced in (Yin et al., 16 Sep 2025) is that NoRA constrains adaptation to a low-dimensional functional subspace. First-order changes from activation updates lie in the span of activation-parameter gradients propagated through the frozen network, with intrinsic dimension at most the sum of activation parameter counts. The paper presents this as an implicit regularization of update magnitude and direction. It also gives a Lipschitz-factorized network bound,

$G$0

and states that a deviation bound shows output changes grow at most linearly, to first order, with activation-parameter updates multiplied by downstream Lipschitz constants (Yin et al., 16 Sep 2025).

The same analysis argues for complementarity between activation-space and weight-space tuning. Specifically, activation-parameter gradients contribute a neural tangent kernel component that is described as largely orthogonal, in expectation, to weight-only directions at initialization. This suggests that the empirical benefit of NoRA++ is not merely additive parameter count, but the use of an additional functional degree of freedom. The approximation argument is also central: rational functions uniformly approximate common activations on bounded ranges, with fast convergence rates for analytic activations and near-root-exponential rates for kinked functions, while the pole-free denominator form maintains bounded slopes and curvature (Yin et al., 16 Sep 2025).

The compute cost of activation-space tuning is reported as modest rather than negligible. On ViT-Tiny, NoRA increases FLOPs from $G$1G for LoRA to $G$2G and latency from $G$3 to $G$4 ms/sample, due to polynomial numerator and denominator evaluation. At the same time, NoRA reduces trainable parameters by more than $G$5 versus LoRA on ViT-Tiny, $G$6K versus $G$7K excluding the classification head. For practical use, the paper recommends replacing GELU or SiLU in MLP blocks, keeping attention and normalization unchanged, using practical group counts $G$8, and employing small ranks such as $G$9 (Yin et al., 16 Sep 2025).

The reported limitations are correspondingly specific. Minor regressions appear in some MMLU categories. Gains depend on a reasonable choice of ranks and degrees; excessive ranks yield diminishing returns, while too few groups can underfit. Future work identified in the paper includes adaptive grouping, dynamic rank and degree selection, alternative functional families such as splines and Fourier-inspired activations, extension of activation-space adapters to normalization layers and generative architectures, and scaling to larger models and more complex tasks including long-context reasoning, multi-hop, and multimodal settings (Yin et al., 16 Sep 2025).