Activation-Aware Fine-Tuning Methods

• Activation-aware fine-tuning is a family of methods that optimize internal activations as explicit adaptation targets, enabling models to adjust nonlinearities for improved performance.
• The paradigm spans various techniques including learnable activation functions, activation steering, and sparsity control, which are applied in full fine-tuning, PEFT, and compression settings.
• These methods provide practical benefits such as enhanced transfer learning, efficient parameter updates, and deeper insights into the internal circuitry of neural networks.

Searching arXiv for the cited papers and closely related activation-aware fine-tuning work. Activation-aware fine-tuning denotes a family of adaptation methods in which internal activations, activation functions, activation distributions, or activation-derived diagnostics are treated as explicit objects of optimization, control, or analysis. In this literature, activations are not merely intermediate tensors needed for backpropagation. They become learnable nonlinearities, targets for alignment, steering variables, sparsity objectives, neuron-selection criteria, low-rank bases, and mechanistic probes of post-training behavior. The resulting paradigm spans full fine-tuning, parameter-efficient fine-tuning (PEFT), reinforcement-learning post-training, activation steering, quantization-aware adaptation, zeroth-order optimization, and even fine-tuning-free compression (Fang et al., 2022, Yin et al., 16 Sep 2025, Zhang et al., 25 Sep 2025, Mishra et al., 26 Sep 2025, Adila et al., 28 Feb 2026, Runwal et al., 2024, Dong et al., 1 Jul 2026, Martínez, 29 Sep 2025).

1. Definition, scope, and historical development

One major lineage of activation-aware fine-tuning begins from the observation that Transformer activation functions are typically fixed from pre-training to fine-tuning. “Transformers with Learnable Activation Functions” introduced Rational Activation Functions (RAFs) into BERT-style feed-forward networks, replacing fixed GELU with a learnable rational form

$$F(x)=\frac{\sum_{j=0}^{m} a_j x^j}{1+\left|\sum_{k=0}^{n} b_k x^k\right|},$$

and showed that the resulting RAFT model improved validation perplexity and downstream transfer, while learned activation shapes varied substantially across layers and tasks (Fang et al., 2022). “Don’t Forget the Nonlinearity: Unlocking Activation Functions in Efficient Fine-Tuning” extended this line into PEFT by proposing NoRA, which freezes weights and adapts nonlinearities directly, and NoRA++, which combines activation-space adaptation with LoRA (Yin et al., 16 Sep 2025).

A second lineage treats activation states rather than activation functions as the main adaptation target. In this view, one does not only alter parameters that produce activations; one also aligns, steers, amplifies, or regularizes the activations themselves. “IA2: Alignment with ICL Activations Improves Supervised Fine-Tuning” uses hidden self-attention outputs under in-context learning as a self-distillation target before ordinary SFT (Mishra et al., 26 Sep 2025). “Weight Updates as Activation Shifts: A Principled Framework for Steering” formalizes a first-order equivalence between activation-space interventions and weight-space updates, arguing that activation steering can locally replicate fine-tuning behavior and that the post-block output is a theoretically preferred intervention locus (Adila et al., 28 Feb 2026).

A third lineage treats activation statistics as mechanistic diagnostics or systems-level constraints. “Reinforcement Learning Fine-Tuning Enhances Activation Intensity and Diversity in the Internal Circuitry of LLMs” measures how RL post-training changes edge attributions in the residual stream graph (Zhang et al., 25 Sep 2025). “From PEFT to DEFT” adds a density loss to reduce activation density in Transformer MLP blocks (Runwal et al., 2024). “RoLoRA” uses rotation-aware fine-tuning to preserve outlier-free activations under weight-activation quantization (Huang et al., 2024). This suggests that activation-aware fine-tuning is best understood not as a single method, but as a common design principle: internal computation becomes an explicit adaptation interface.

2. Learnable nonlinearities as trainable adaptation objects

In the learnable-activation formulation, the key claim is that fine-tuning need not be restricted to updating weight matrices. RAFT retains the BERT encoder architecture and replaces the FFN nonlinearity in

$\mathrm{FFN}(x)=W_2\,\phi(W_1x)$

with a rational activation, yielding

$\mathrm{FFN}(x)=W_2\,F(W_1x).$

The RAF coefficients are learned end-to-end with the masked language modeling objective. On the reported setup, RAFT improved validation perplexity from 5.18 to 5.00 and improved GLUE averages in both low-data and full-data settings, with especially strong gains when only 100 examples were available (Fang et al., 2022).

NoRA generalizes this activation-centric view to PEFT. It models each trainable activation as a rational function

$$\phi(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{m} a_i x^i}{\sum_{j=0}^{n} b_j x^j},$$

and uses the stabilized form

$$\phi(x)=\frac{a_0+a_1x+\cdots+a_mx^m}{1+\left|b_0+b_1x+\cdots+b_nx^n\right|}.$$

Rather than updating all coefficients freely, NoRA applies structured low-rank updates to numerator and denominator coefficients with group-wise sharing. The activation for group $g$ becomes

$$\phi'_g(X)=\frac{(P_g+A^P_gB^P_g)(X)}{(Q_g+A^Q_gB^Q_g)(X)},$$

with low-rank factors initialized LoRA-style so that the pretrained activation is exactly recovered at initialization. The paper argues that numerator updates shape the function, denominator updates control saturation and numerical stability, and joint perturbation of both is empirically best (Yin et al., 16 Sep 2025).

The group-wise design is central. Instead of one rational activation per neuron, hidden channels are partitioned into $G$ disjoint groups sharing activation functions. This is described as a compromise between expressivity and efficiency, and the paper reports that increasing the number of groups improves accuracy while parameter growth remains modest. On ViT-Tiny pretrained on ImageNet-1K, NoRA updates only 0.02M parameters, about 0.4% of the model, yet exceeds full fine-tuning on CIFAR-10 and CIFAR-100; NoRA++ achieves the strongest overall accuracy under matched budgets. On LLaMA3-8B instruction tuning, NoRA++ improves average MMLU by about $+0.3\%$ to $+0.8\%$ over LoRA and yields larger STEM gains on several datasets (Yin et al., 16 Sep 2025).

These results are accompanied by a function-space interpretation. The first-order functional change induced by activation updates lies in a low-dimensional span, and tuning activations changes the Lipschitz constant, Jacobian spectrum, and Hessian or curvature without changing $\mathrm{FFN}(x)=W_2\,\phi(W_1x)$0. A plausible implication is that activation-aware PEFT imposes a different inductive bias from weight-centric PEFT: it constrains adaptation through the nonlinear transfer functions that mediate representation flow, rather than only through linear maps (Yin et al., 16 Sep 2025).

3. Mechanistic diagnostics of internal circuitry

A separate body of work defines activation-aware fine-tuning through mechanistic analysis of how post-training alters internal computation. In “Reinforcement Learning Fine-Tuning Enhances Activation Intensity and Diversity in the Internal Circuitry of LLMs,” the residual stream is treated as a directed acyclic graph in which attention blocks and FFN blocks are nodes and residual contributions are edges. For an edge $\mathrm{FFN}(x)=W_2\,\phi(W_1x)$1, the paper replaces expensive ablation-based importance with the Edge Attribution Patching approximation

$\mathrm{FFN}(x)=W_2\,\phi(W_1x)$2

This permits large-scale comparison of edge scores across datasets, model pairs, and truncation settings (Zhang et al., 25 Sep 2025).

Using the edge-weight tensor $\mathrm{FFN}(x)=W_2\,\phi(W_1x)$3, the paper defines three dataset-level metrics: Activation Intensity, the mean absolute edge attribution; Information Complexity, an entropy-style measure over absolute edge weights; and Distribution Kurtosis, which measures how concentrated or heavy-tailed the per-sample edge distribution is. Across DeepSeek-Math, Mistral, and Distilled-Qwen model families, online RL post-training generally increases activation intensity and information complexity while decreasing distribution kurtosis. Pairwise correlation between edge-weight matrices across samples also becomes lower after RL, and output-edge entropy increases at many nodes. The authors interpret this as evidence that RL engages more internal pathways and distributes computation more broadly across them (Zhang et al., 25 Sep 2025).

The main methodological contrast is between PPO or GRPO and DPO. Qwen2.5-7B-DPO does not show the same stable pattern: changes in activation intensity, complexity, and kurtosis are weaker and less regular. The paper attributes this to the fact that DPO uses a fixed set of preference pairs sampled once from the SFT policy, whereas PPO and GRPO operate on sampled responses from the current policy $\mathrm{FFN}(x)=W_2\,\phi(W_1x)$4. The stated interpretation is that DPO’s static supervision cannot reliably expand active internal circuitry in the way online RL can (Zhang et al., 25 Sep 2025).

IA2 extends the diagnostic viewpoint from RL to the comparison between in-context learning and supervised fine-tuning. It defines an activation tensor $\mathrm{FFN}(x)=W_2\,\phi(W_1x)$5 from hidden self-attention outputs and uses token-wise cosine similarity to compare activation trajectories. The reported finding is that ICL and SFT activations are poorly aligned, especially in middle layers, even when trained on the same few-shot task data. IA2 then uses an activation-matching loss

$\mathrm{FFN}(x)=W_2\,\phi(W_1x)$6

during a priming stage before ordinary SFT, with the stated goal of replicating ICL’s activation patterns and incentivizing ICL-like internal reasoning (Mishra et al., 26 Sep 2025). This suggests that activation-aware fine-tuning can operate not only by changing which circuits are active, but also by specifying which internal trajectories are desirable.

4. Activation steering, alignment, and inference-time control

Activation-aware adaptation is not limited to parameter updates. Several methods intervene directly in activation space during inference or use activation-space objectives to guide later training. “Weight Updates as Activation Shifts” shows that for an MLP $\mathrm{FFN}(x)=W_2\,\phi(W_1x)$7, a small perturbation to the input activation yields

$\mathrm{FFN}(x)=W_2\,\phi(W_1x)$8

The paper uses this first-order bridge to argue that activation steering can locally mimic fine-tuning, explains why different intervention sites have different expressivity, and identifies the post-block output as the preferred locus because it acts on the full residual stream after attention, MLP, and skip connection. It also introduces joint adaptation, combining a LoRA-style weight update with an activation adapter under an orthogonality constraint so that the two spaces do not collapse into the same dominant directions (Adila et al., 28 Feb 2026).

Inference-time activation engineering provides concrete examples of this perspective. “Temporal Alignment of Time Sensitive Facts with Activation Engineering” injects steering vectors built from temporal prompts into the residual stream of LLaMA 2 models, with no weight updates and no pre-aligned dataset. The procedure extracts activations from steering prompts, scales them by coefficients, sums them, and adds the resulting vector at selected layers of a user prompt. The paper reports up to a 44% improvement in relative prompting and 16% in explicit prompting, and states that performance is comparable to or sometimes better than the reported fine-tuning baseline on temporal QA benchmarks (Govindan et al., 20 May 2025).

“Activation Control for Efficiently Eliciting Long Chain-of-thought Ability of LLMs” likewise begins from an activation-level hypothesis: a small set of high-impact activations in the last few layers largely governs output length, self-reflection, and long-form reasoning. The training-free method identifies top activations from contrastive examples, inserts “wait” tokens, and amplifies selected activations according to

$\mathrm{FFN}(x)=W_2\,\phi(W_1x)$9

with reported gains in self-reflection and reasoning accuracy. The same mechanistic insight is then converted into a PEFT method that trains only a last-layer activation amplification module and a few LoRA layers, using 1.51% of parameters versus 6.15% for the reported LoRA baseline (Zhao et al., 23 May 2025).

A common misconception is that activation-aware methods are identical with training-free activation engineering. The literature does not support that equivalence. Some methods intervene only at inference time; others fine-tune learnable activation functions, match activation trajectories, or optimize weight updates constrained by activation-derived subspaces (Govindan et al., 20 May 2025, Zhao et al., 23 May 2025, Yin et al., 16 Sep 2025, Mishra et al., 26 Sep 2025, Adila et al., 28 Feb 2026).

5. Efficiency, sparsity, quantization, and low-memory optimization

A large subset of activation-aware fine-tuning addresses systems constraints by controlling activation density, activation outliers, activation memory, or activation-informed update spaces. DEFT augments standard PEFT with a density loss over intermediate MLP activations,

$\mathrm{FFN}(x)=W_2\,F(W_1x).$0

where the backbone $\mathrm{FFN}(x)=W_2\,F(W_1x).$1 is frozen and only PEFT parameters $\mathrm{FFN}(x)=W_2\,F(W_1x).$2 are trained. The objective is to reduce activation density, equivalently increase activation sparsity, so that sparsity-aware hardware can skip zero-valued computations. The paper reports activation-density reductions of up to 44.94% on RoBERTa-Large and 53.19% encoder density plus 90.60% decoder density on Flan-T5-XXL, together with runtime and memory savings in ADA-DEFT (Runwal et al., 2024).

RoLoRA addresses a different failure mode: activation outliers under low-bit weight-activation quantization. It first applies orthogonal or Hadamard rotations to eliminate outliers, then performs rotation-aware LoRA fine-tuning in the rotated space so the outlier-free property is preserved. Activation kurtosis is used as the outlier metric, and the paper reports large gains in W4A4 GPTQ settings, including a 29.5-point improvement in zero-shot commonsense reasoning average on LLaMA2-13B compared with the LoRA baseline (Huang et al., 2024).

TokenSeek targets activation memory rather than activation values themselves. It scores tokens instance-wise using both attention-derived context importance and penultimate-layer gradient importance,

$\mathrm{FFN}(x)=W_2\,F(W_1x).$3

then computes gradients only for selected tokens while disabling gradient tracking for the rest. The default reported setting keeps 10% of tokens. On several models, the paper reports large training-memory reductions, such as 14.8% of the memory on Llama3.2 1B in the QLoRA + TokenSeek setting, with on-par or better performance than full-token tuning (Zeng et al., 27 Jan 2026).

ZO-Act moves the activation-aware idea into forward-only optimization. For each linear layer, it collects input activations once at initialization, computes the top right singular vectors $\mathrm{FFN}(x)=W_2\,F(W_1x).$4, freezes them, and parameterizes the update as

$\mathrm{FFN}(x)=W_2\,F(W_1x).$5

Only the small coefficient matrix $\mathrm{FFN}(x)=W_2\,F(W_1x).$6 is trained using forward-difference loss evaluations. The paper argues that this reduces zeroth-order variance because optimization occurs in a coefficient space of dimension $\mathrm{FFN}(x)=W_2\,F(W_1x).$7, not the full weight space, and reports consistent gains over strong zeroth-order baselines on Llama-3-8B, OPT-13B, and INT4 Llama-3-8B (Dong et al., 1 Jul 2026).

In diffusion models, activation-awareness is coupled with temporal structure. “Pioneering 4-Bit FP Quantization for Diffusion Models” identifies asymmetric activation distributions induced by SiLU, proposes mixup-sign floating-point quantization with unsigned FP plus zero point for anomalous layers, and introduces Timestep-Aware LoRA together with denoising-factor loss alignment

$\mathrm{FFN}(x)=W_2\,F(W_1x).$8

The paper reports that the full combination of MSFP, TALoRA, and DFA achieves the best 4-bit results among the reported ablations (Zhao et al., 27 May 2025). Across these methods, activations function simultaneously as a bottleneck, a regularization target, and a structural prior for efficient optimization.

6. Neuron specialization, domain-specific adaptation, and broader implications

Activation-aware fine-tuning also appears in neuron-selective and domain-specific form. MNAFT, for multimodal image translation, identifies language-agnostic and language-specific neurons in both vision and language modules through instruction-driven activation analysis. Neuron importance is estimated by a first-order Taylor criterion,

$\mathrm{FFN}(x)=W_2\,F(W_1x).$9

and only the parameters corresponding to selected general and specific neurons in selected layers are updated through a gradient mask. On six translation tasks across four datasets, the paper reports that MNAFT outperforms cascaded systems, full fine-tuning, and several PEFT baselines, while the neuron-identification stage takes about 2–3 minutes (Li et al., 18 Apr 2026).

Even compression work participates in the same conceptual shift. BALF is explicitly fine-tuning-free, but it is activation-aware because it replaces parameter-reconstruction objectives with empirical activation distortion,

$$\phi(x)=\frac{P(x)}{Q(x)}=\frac{\sum_{i=0}^{m} a_i x^i}{\sum_{j=0}^{n} b_j x^j},$$0

uses calibration activations to whiten layers before low-rank truncation, and allocates ranks globally under a FLOP or parameter budget. The paper reports, for example, that BALF reduces FLOPs on ResNeXt-101 by 45% with only a 1-percentage-point top-1 accuracy drop, without fine-tuning (Martínez, 29 Sep 2025).

Several broader implications recur across the literature. First, output-level evaluation alone is treated as insufficient when the aim is to understand how adaptation works internally; RL probing, ICL-to-SFT activation alignment, and neuron-aware multimodal tuning all treat hidden computation as a first-class object (Zhang et al., 25 Sep 2025, Mishra et al., 26 Sep 2025, Li et al., 18 Apr 2026). Second, the distinction between weight-space and activation-space adaptation is increasingly explicit: some methods argue for complementarity rather than substitution, as in NoRA++ and joint adaptation with orthogonality constraints (Yin et al., 16 Sep 2025, Adila et al., 28 Feb 2026). Third, activation-aware methods are not uniformly training-based. Some are training-free inference-time interventions, some are PEFT schemes, some modify nonlinearities, and some are calibration-only compression procedures (Govindan et al., 20 May 2025, Martínez, 29 Sep 2025).

This suggests a stable editorial summary of the field. Activation-aware fine-tuning is the study and practice of adapting models by explicitly reasoning about internal activations: their shape, location, sparsity, density, outlier structure, trajectory, entropy, or task-specific salience. In the strongest form, it treats internal computation not merely as a consequence of optimization, but as a design target in its own right.