Decomposing the Depth Profile of Fine-Tuning

Abstract: Fine-tuning adapts pretrained networks to new objectives. Whether the resulting depth profile of representational change reflects an intrinsic property of the model or the magnitude of gradient flow has not been tested directly. We measure this profile across 240 fine-tuning runs spanning 15 models in four architecture families (encoder and decoder transformers, a state-space model, and an RNN) at scales from 125M to 6.9B parameters. Representational change concentrates in output-proximal layers in every standard-training run except one. We apply a per-layer control that equalizes $|ΔW|/|W|$ across layers after each optimizer step. Under this control, the profile persists in some conditions and collapses in others. At 125M--350M, sequential-block architectures (BERT, OPT, GPT-2) retain the slope across tested objectives while parallel-block architectures (Pythia, CodeGen) retain it only for causal-language-modeling objectives. This architectural distinction narrows at 1.3B--1.4B, where both block types show positive equal-step slopes for CausalLM. Under standard training, profile shape is described by two additional axes: steepness tracks a training-free objective distance at initialization, and profile width is dominated by architecture. We treat the locality gradient, the depthwise slope of representational change, as a composite phenomenon whose components are scale-dependent.

• The paper introduces an equal-step control to separate intrinsic representational change from optimizer-induced effects.
• It empirically quantifies depth profiles across 15 models using Procrustes and cosine metrics, revealing clear architecture-specific adaptation patterns.
• Findings challenge conventional PEFT heuristics, highlighting that depth-targeted tuning must be adapted to model architecture, objective, and scale.

Decomposing the Depth Profile of Fine-Tuning: Disentangling Representational Change Across Network Depths

Introduction

This paper addresses a central, yet under-explored, phenomenon in transfer and adaptation of pretrained neural networks: the distribution of representational change across network depth during fine-tuning. While empirical heuristics (e.g., freezing lower layers, late-layer LoRA/adapters, and depth-dependent LR scheduling) have become standard, their theoretical justification—beyond observation in select transformer models—has lacked systematic, broad-based evaluation. The paper seeks precise characterization and causal dissection of the depth profile (the "locality gradient") of representational change due to fine-tuning across various architectures, objectives, and conditions. Its methodological innovation is the introduction of a per-layer "equal-step" (trust-ratio) control that neutralizes native gradient magnitude asymmetries, thus dissociating representational restructuring arising from intrinsic model properties versus mere optimizer dynamics.

Methodology

Experimental Scope

The study encompasses 240 fine-tuning runs over 15 models, spanning four architecture classes—encoder transformers, decoder transformers (with both sequential and parallel residual block variants), state-space models (Mamba), and RNNs (RWKV)—ranging from 125M to 6.9B parameters. For each model-objective pairing, changes are quantified at seven normalized depth positions using both Procrustes distance and linear CKA on activation matrices sampled from a standardized probe set.

Equal-Step Control

The equal-step condition forces per-layer update norms ($\|\Delta W_l\|/\|W_l\|$) to a constant $\tau$ after each optimizer step, analogous to the trust-ratio normalization of LARS/LAMB but used diagnostically (not as a regularizer). This isolates the role of depth-dependent gradient magnitude in shaping the post-finetuning representational profile.

Objective Distance Metric

A training-free, model-conditioned measure of task distance is calculated from the gradient (w.r.t. last-layer representation) induced by each objective at initialization, using Procrustes and cosine metrics, providing a theoretical lower-bound for representational displacement.

Main Results

Universality and Non-Triviality of the Locality Gradient

Representational change is robustly output-localized in standard fine-tuning runs: in 189/190 cases, both distance measures yield a profile increasing towards later layers across all architectures and scales (to 6.9B). Importantly, this phenomenon is not solely a function of gradient magnitude, as demonstrated by the equal-step control.

Disentangling Architecture and Objective Effects

• Sequential-Block Decoders (e.g. GPT-2, OPT): Output-localization of change persists—and can even steepen—under equal-step normalization, indicating intrinsic depth-structured adaptation.
• Parallel-Block Decoders (e.g. Pythia, CodeGen): The locality gradient collapses under equal-step for most objective families, except for CausalLM-type objectives at larger scales (≥1.4B) where positive slopes re-emerge, indicating emergent intrinsic structure.
• Contrastive Objectives: These (especially SimCSE, BarlowTwins) tend to drive collapse of the locality gradient under equal-step in causal models, even when standard fine-tuning displays output-localization, reflecting objective-driven fragility rather than intrinsic model property.

Quantification of Profile Structure

• Steepness (Slope): Strongly tracks training-free objective distance (8/9 positive model-level correlations under Procrustes, 9/9 under cosine), confirming the proportionality between objective disparity and required representational restructuring.
• Width (Spread): Dominated by architectural class. Sequential-block decoders focus $>$98% of total representational change in the final layer. In contrast, parallel-block decoders and encoders spread adaptation more broadly (final layers account for only 37–57%), with state-space and RNN models intermediate.

Scale Effects

At higher parameter counts (1.3–1.4B), the architecture-dependence of the locality gradient weakens for CausalLM objectives: both sequential and parallel decoders display significant positive slopes under equal-step.

Discussion and Implications

The findings directly challenge the generalization of BERT-derived heuristics, which undergird a significant fraction of PEFT (e.g., LoRA/adapters placement, layer freezing) across the LLM engineering landscape. The output-localization effect is empirically robust but mechanistically composite—partially arising from gradient flow artifacts, partially from intrinsic network feature routing, and substantially contingent on architecture, objective family, and scale. Consequently:

• Depth-Dependent Adaptation recipes (late-layer LoRA, bias-only tuning, etc.) are only justifiable in sequential-block decoders and for certain objectives at scale in parallel-block decoders. Otherwise, these methods may exploit optimizer artifacts rather than intrinsic feature hierarchy.
• Equal-step is proposed as a fast diagnostic: a single controlled fine-tuning run suffices to disambiguate whether a given (model, objective) pairing admits meaningful intrinsic depth structure exploitable by depth-targeted adaptation.
• Model architecture is the principal determinant of adaptation width, with little contribution from the fine-tuning objective or scale (outside CausalLM at ≥1.4B).
• Objective distance metrics, computable pre-training, can predict required adaptation magnitude and possibly inform PEFT parameter allocation and placement priori.

Limitations

The study's equal-step control operates in weight space, not functional/Fisher space; the translation of these observations to functionally-weighted parameter reallocation is unaddressed. The coverage of state-space and RNN models is limited, as are the tested objectives (no RLHF or multimodal tasks). Block-structure and scale causal separation would benefit from ablations with more directly matched pretraining and hyperparameters. Mamba/RWKV, while informative, are single-model representatives.

Conclusion

The depth profile of fine-tuning-induced representational change is a robust but composite phenomenon. Its mechanistic drivers depend on intricate interactions between model architecture, training objective, and scale regime. PEFT strategies based on output-localized adaptation must be re-justified in non-sequential decoders and non-CausalLM objectives, as equal-step analysis exposes the contribution of optimizer artifacts. The practical upshot is two-fold: empirically, layer-targeting and adaptation allocation must be model/goal/scale conditional; theoretically, the "locality gradient" is best understood as an emergent, multifactorial signal, not a universal law.

Reference: "Decomposing the Depth Profile of Fine-Tuning" (2604.17177)