Encoder-Based Classification Fine-Tuning

• Encoder-based classification fine-tuning is a method that adapts pre-trained encoders with lightweight, task-specific heads using pooled representations like the [CLS] token.
• It employs diverse strategies including full end-to-end tuning, frozen and LoRA-based updates, while integrating various loss functions such as cross-entropy and contrastive losses.
• Careful optimization of pooling methods, learning rates, and normalization techniques has demonstrated significant improvements across modalities, achieving competitive metrics in text, imaging, and audio tasks.

Encoder-based classification fine-tuning is the adaptation of a pre-trained encoder to a supervised classification objective by attaching a task-specific head to a pooled representation and then updating some or all model parameters on downstream data. In BERT-style sequence classification, a common practice is to feed the embedding of the final-layer [CLS] token to a classification layer and fine tune the parameters of the encoder and classifier jointly (Zhou et al., 2019). Across the literature, this pattern appears in encoder-only NLP models such as BERT, RoBERTa, SciBERT, CamemBERT, mBERT, XLM-R, and BERTurk, and also in non-text encoders in medical imaging, audio, and multimodal image-text systems, where pooled latent features are mapped to class logits by linear or lightweight adapter heads (Sun et al., 2019, Zhao et al., 2024, Mouheb et al., 29 Aug 2025).

1. Canonical architectural pattern

The most common formulation uses a pre-trained encoder to map an input sequence to contextual hidden states and then applies a shallow classification head to a pooled representation. For BERT fine-tuning, the input is tokenized with WordPiece, prefixed by [CLS], suffixed or segment-split by [SEP], and processed by 12 Transformer blocks; the final hidden state at the [CLS] position, $h \in \mathbb{R}^{768}$, is passed through a single linear layer and softmax, $p(c \mid h)=\mathrm{softmax}(Wh+b)$ (Sun et al., 2019). RoBERTa fine-tuning in single-task text classification follows the same pattern, with a sequence head $W_{\mathrm{seq}} \in \mathbb{R}^{K \times D}$ and bias $b_{\mathrm{seq}}$, $$\hat y_i=\mathrm{softmax}(W_{\mathrm{seq}} h_{(\mathrm{CLS})}+b_{\mathrm{seq}})$$, while token classification uses per-token hidden vectors and a separate token head (Zhao et al., 2024).

Binary and multi-label variants retain the encoder and replace the final activation. In German web topic detection, a single feed-forward layer is appended to the [CLS] embedding, with architecture Dropout(p = 0.1) → Linear(H→1) → Sigmoid, producing a probability for the positive class (Schelb et al., 2024). In French plant-health bulletin classification, CamemBERT-base and multilingual BERT use the final hidden state of [CLS], a fully connected layer $W\in\mathbb{R}^{H\times 2}$, and an element-wise sigmoid to produce independent probabilities for the pest and disease labels (Jiang et al., 2021). BERTurk text classification uses [CLS], dropout $(p=0.1)$, a dense layer, and softmax over $C$ logits (Yildirim, 2024).

Multi-task encoder fine-tuning extends the same design by attaching several heads to one shared encoder. In astronomy knowledge extraction, SciBERT is fine-tuned with one multiclass telescope head and four independent binary heads, all fed by the shared final [CLS] representation $h_{\text{CLS}}\in\mathbb{R}^{H}$ with $H=768$ (Rawat et al., 11 Nov 2025). In RoBERTa-based MASSIVE experiments, intent detection uses a sequence head over [CLS], while slot filling uses a token head over per-token hidden states, with all RoBERTa layers shared (Zhao et al., 2024).

Outside text, the architectural template persists but the pooling operator changes. In federated MRI-based dementia classification, the SAM-Med3D encoder produces a latent tensor $p(c \mid h)=\mathrm{softmax}(Wh+b)$0; a linear head performs global average pooling to $p(c \mid h)=\mathrm{softmax}(Wh+b)$1 and then applies a single linear layer plus softmax, while larger alternatives add small 3D convolutional adapters before the classifier (Mouheb et al., 29 Aug 2025). In cough classification, Whisper-tiny’s decoder is discarded, the encoder output is pooled by active-frame QKV attention over the first $p(c \mid h)=\mathrm{softmax}(Wh+b)$2 tokens, and the pooled representation is sent to the disease head (Vincent, 2 Jun 2026).

2. Loss functions and prediction regimes

The dominant supervised objective is cross-entropy or binary cross-entropy, with straightforward extensions to token labeling, multi-task learning, and multi-label prediction. For single-task sequence classification in RoBERTa, the loss is

$p(c \mid h)=\mathrm{softmax}(Wh+b)$3

and slot filling uses token-wise cross-entropy,

$p(c \mid h)=\mathrm{softmax}(Wh+b)$4

In multi-task settings, the cited work simply sums the losses, $p(c \mid h)=\mathrm{softmax}(Wh+b)$5 (Zhao et al., 2024).

Binary classification tasks use the standard binary cross-entropy. In German webpage topic detection,

$p(c \mid h)=\mathrm{softmax}(Wh+b)$6

with early stopping on validation F1 and regularization through dropout and weight decay (Schelb et al., 2024). In French plant-health bulletin classification, the objective sums binary cross-entropy over the two labels, and the encoder is fine-tuned end-to-end with Adam (Jiang et al., 2021). Astronomy knowledge extraction combines a multiclass cross-entropy for telescope prediction with four binary cross-entropy losses for the Boolean labels, with default equal weights $p(c \mid h)=\mathrm{softmax}(Wh+b)$7 and $p(c \mid h)=\mathrm{softmax}(Wh+b)$8 (Rawat et al., 11 Nov 2025).

Auxiliary objectives are used when plain supervised loss is considered insufficient. In lightweight transformer pre-finetuning for text classification and NER, the text-classification branch uses an InfoNCE contrastive loss with temperature $p(c \mid h)=\mathrm{softmax}(Wh+b)$9, the NER branch uses cross-entropy, and the multi-task pre-finetuning objective is $W_{\mathrm{seq}} \in \mathbb{R}^{K \times D}$0 (Zhu et al., 8 Oct 2025). In cough classification, the total batch loss is

$W_{\mathrm{seq}} \in \mathbb{R}^{K \times D}$1

combining focal loss, a domain-adversarial branch with gradient reversal, and supervised contrastive loss (Vincent, 2 Jun 2026). In zero-shot X-ray pathology classification, contrastive image-text encoders are fine-tuned with a modified CLIP-style InfoNCE objective, augmented by positive-pair loss relaxation and random sentence sampling (Jang et al., 2022).

These formulations show that encoder-based classification fine-tuning is not restricted to single-label softmax classification. The same encoder can support binary, multi-class, multi-label, token-level, contrastive, and hybrid objectives, provided that the head and loss are matched to the prediction regime.

3. Parameter-update strategies and optimization control

The cited work describes a wide range of update regimes: full end-to-end fine-tuning, linear probing with a frozen encoder, parameter-efficient tuning, staged unfreezing, and hierarchical freezing. Classical BERT fine-tuning updates all encoder layers plus the classifier, but several studies report that careful control of learning rates and parameter movement is important. In the exhaustive BERT text-classification study, aggressive learning rates $W_{\mathrm{seq}} \in \mathbb{R}^{K \times D}$2 cause rapid degradation of pre-trained features, whereas a base learning rate of $W_{\mathrm{seq}} \in \mathbb{R}^{K \times D}$3 with warm-up is low enough to preserve knowledge yet flexible enough to adapt; layer-wise learning-rate decay with $W_{\mathrm{seq}} \in \mathbb{R}^{K \times D}$4 produced the best IMDb error among the reported settings (Sun et al., 2019).

Optimization can also be improved by intervening on the pooled representation. A systematic analysis of BERT sequence classification reported a biased embedding distribution issue before fine tuning: embedding values of [CLS] concentrate on a few dimensions and are non-zero centered, which brings challenge to the optimization process during fine-tuning because gradients of [CLS] embedding may explode and result in degraded model performance. The paper proposes several simple normalization methods to modify the [CLS] embedding during fine-tuning, and the normalized embedding shows improvements on several text classification tasks (Zhou et al., 2019).

Selective tuning is often motivated by efficiency or data scarcity. In federated MRI-based dementia classification, full fine-tuning, encoder-frozen fine-tuning, and LoRA all achieved $W_{\mathrm{seq}} \in \mathbb{R}^{K \times D}$5 AUC on the pooled test set with overlapping confidence intervals, while freezing the backbone reduced communication by approximately $W_{\mathrm{seq}} \in \mathbb{R}^{K \times D}$6 and GPU memory by approximately $W_{\mathrm{seq}} \in \mathbb{R}^{K \times D}$7 (Mouheb et al., 29 Aug 2025). In hierarchical fine-tuning for liver lesion classification, transferring encoder-decoder weights from LiTS and then progressively freezing or unfreezing encoder blocks improved both Dice and classification accuracy relative to training from scratch, with hierarchical unfreezing reaching accuracy $W_{\mathrm{seq}} \in \mathbb{R}^{K \times D}$8 in 3-fold cross-validation (Heker et al., 2019). In CoughSense, training is explicitly two-phase: the entire Whisper encoder is frozen for epochs 1–3 and then unfrozen for epochs 4–25 with differential learning rates, $W_{\mathrm{seq}} \in \mathbb{R}^{K \times D}$9 and $b_{\mathrm{seq}}$0 (Vincent, 2 Jun 2026).

Parameter-efficient tuning introduces modularity rather than simply reducing trainable parameter count. In multi-task pre-finetuning of lightweight encoders, MTPF-TPL attaches two LoRA modules to the last $b_{\mathrm{seq}}$1 transformer layers: LoRA_TC is updated only with the contrastive text-classification loss, LoRA_NER only with the NER loss, while the shared backbone $b_{\mathrm{seq}}$2 is updated with both losses (Zhu et al., 8 Oct 2025). This design directly addresses the reported problem that naïve multi-task pre-finetuning introduces conflicting optimization signals that degrade overall performance (Zhu et al., 8 Oct 2025).

A further qualification concerns what is often described as catastrophic forgetting. Fine-tuning on a subset of classes can drastically degrade accuracy on absent classes, but a detailed analysis argues that the fine-tuned model often produces more discriminative features for those absent classes; the main problem is discrepant logit scales between fine-tuning classes and absent classes. An additive post-processing calibration,

$b_{\mathrm{seq}}$3

recovers and often exceeds pre-trained performance on absent classes (Mai et al., 2024). This suggests that poor post-fine-tuning behavior is not always evidence that encoder features have been destroyed.

4. Input construction, truncation, pooling, and aggregation

A central engineering problem in encoder-based classification fine-tuning is the mismatch between input length and encoder context window. The basic BERT recipe uses a maximum length of 512 sub-word tokens (Sun et al., 2019). For long documents, truncation strategy matters: on IMDb and Sogou, keeping the head and tail of the document outperformed head-only, tail-only, or hierarchical pooling, with reported error rates of 5.42 / 2.43 for head+tail versus 5.63 / 2.58 for head-only and 5.44 / 3.17 for tail-only (Sun et al., 2019).

When truncation is too lossy, document chunking and aggregation are used. In astronomy papers that exceed SciBERT’s 512-token limit, each document is split into non-overlapping 512-token chunks; during training, up to $b_{\mathrm{seq}}$4 chunks per document are sampled uniformly at random, and at test time all segments are processed and aggregated by majority voting (Rawat et al., 11 Nov 2025). The reported Macro-F1 improves from 0.72 for SciBERT_v1 (first 512 tokens only) to 0.73 for SciBERT_v2 (10 random chunks), and the discussion attributes the gain to varied parts of the paper, particularly acknowledgments and grant sections (Rawat et al., 11 Nov 2025).

German web topic classification uses a related but distinct strategy. Content is split into overlapping chunks with 384 tokens max, 64-token overlap, and each chunk inherits the URL label; the framework also compares URL only, Content only, and URL + Content feature constructions (Schelb et al., 2024). In chest X-ray report fine-tuning, random sentence sampling selects $b_{\mathrm{seq}}$5 sentences from the report in each training iteration to create alternative positive text views without introducing explicit pathology labels (Jang et al., 2022).

Signal dilution can arise even when the formal context window is sufficient. CoughSense observes that a 3-second cough yields only approximately 150 non-zero tokens out of Whisper’s 1500-token, 30-second input window. It therefore restricts attention pooling to the first $b_{\mathrm{seq}}$6 tokens and replaces mean pooling with learned QKV attention pooling. In the ablation, mean-pooling all 1500 tokens gives 73.1 % balanced accuracy, active-frame cropping raises it to 78.2 %, and adding QKV attention pooling raises it to 80.4 % (Vincent, 2 Jun 2026). This suggests that input aggregation is often as consequential as the encoder backbone itself.

5. Pre-finetuning, continued pre-training, and multi-task organization

Encoder-based classification fine-tuning is frequently preceded by additional unsupervised or weakly supervised adaptation. In BERT text classification, continued masked-LM and next-sentence pre-training on unlabeled data can be performed within-task, in-domain, or cross-domain; in-domain pre-training produced the largest average error reduction over vanilla BERT fine-tuning, approximately 18.6%, compared with approximately 16% for within-task and approximately 14.4% for cross-domain pre-training (Sun et al., 2019). In plant-health bulletin classification, the supervised fine-tuning stage is preceded by domain-adaptive pre-training on unlabeled XML BSV paragraphs for 2 epochs, with batch size 8, initial learning rate 1×10⁻⁴, and optimizer Adam (Jiang et al., 2021). In production query-title relevance classification, domain-specific pre-training of eBERT on BooksCorpus, Wikipedia, and 1 B item titles improves the downstream teacher from ROC AUC=82.26 % for off-the-shelf BERT-base to ROC AUC=87.05 for eBERT-base (Dahlmann et al., 2021).

Pre-finetuning can also be organized as a bridge between task families. MTPF-TPL jointly pre-finetunes a single lightweight encoder for a text-classification contrastive objective and a pseudo-labeled NER objective using task-primary LoRA modules, and then uses the shared encoder plus classification adapter for downstream text classification (Zhu et al., 8 Oct 2025). The reported average accuracy rises from 55.1% for base MiniLM to 63.9% for pre-finetuning on text classification only and 64.1% for MTPF-TPL (Zhu et al., 8 Oct 2025).

Multi-task supervised fine-tuning typically shares the encoder and keeps heads separate. RoBERTa fine-tuning on MASSIVE uses one shared encoder with one sequence head for intent and one token head for slot filling, optimized by the unweighted sum $b_{\mathrm{seq}}$7 (Zhao et al., 2024). Astronomy knowledge extraction similarly shares all SciBERT layers across one multiclass head and four Boolean heads, again with equal weighting (Rawat et al., 11 Nov 2025). The advantage is consolidation: one encoder handles related classification tasks in a single pass.

The supervision source need not be manual labels. For monolingual and zero-shot polylingual neural topic modeling, one strategy is to run LDA on the target corpus, label each document by its most probable LDA topic, and fine-tune the encoder on that topic classification task using cross-entropy (Mueller et al., 2021). The same work also integrates the topic classification objective directly into the contextualized topic model, with

$b_{\mathrm{seq}}$8

The reported findings distinguish objectives by downstream effect: in-domain topic classification gives the highest monolingual NPMI, while NLI fine-tuning gives the best cross-lingual transfer (Mueller et al., 2021).

6. Empirical behavior, trade-offs, and recurrent debates

Reported outcomes vary by task, modality, and evaluation protocol, but several recurring patterns are visible. Encoder-only fine-tuning remains a strong and inexpensive baseline, yet it is not uniformly dominant against larger decoder LLMs. On 20 Newsgroups and MASSIVE, single-task RoBERTa-large reaches 88.1% on 20NG twenty-class, 94.2% on 20NG seven-class, 90.1% on MASSIVE intent, and 85.0% token-level F1 on MASSIVE slot filling, while fully fine-tuned Llama3-70B is reported to outperform RoBERTa-large by approximately 4–5 percentage points; at the same time, RoBERTa large still a strong baseline when compute or data for LLM fine-tuning is limited (Zhao et al., 2024).

Under noisy, unbalanced, and domain-specific conditions, encoder fine-tuning often outperforms alternatives that look competitive on cleaner evaluations. In German policy-topic detection, GELECTRA-Large with URL + Content reaches F1 = 0.430 (best) on the complete test, the average URL only result is 0.037, and URL + Content gives average F1=0.308, a + 40.8% rel. gain over URL-only. The same study states that fine-tuned encoders consistently outperform in-context learning under realistic unbalanced and noisy conditions, even though few-shot Vicuna performs strongly on the balanced test (Schelb et al., 2024).

Domain specialization repeatedly matters. In French plant-health bulletins, multilingual BERT slightly outperforms CamemBERT with Weighted F₁=0.809 and ROC AUC≈0.911 (Jiang et al., 2021). In astronomy knowledge extraction, SciBERT_v2 (10 random chunks) reaches 0.73 Macro-F1 on the held-out test set, compared with 0.31 for OpenAI-GPT-OSS-20B (zero/few-shot) (Rawat et al., 11 Nov 2025). In chest X-ray zero-shot pathology classification, the proposed fine-tuning strategy improves overall zero-shot pathology classification across four datasets and three pre-trained models, with an average macro AUROC increase of 4.3% reported in the abstract (Jang et al., 2022).

Efficiency and deployment constraints frequently favor partial tuning or encoder reuse. In federated MRI-based dementia classification, Encoder-frozen, Full fine-tuning, and LoRA All all report 0.86 AUC, while encoder freezing reduces communication overhead and memory usage (Mouheb et al., 29 Aug 2025). In production query-title relevance classification, eBERT fine-tuning provides the teacher, and a distilled BertBiLSTM student achieves ROC AUC=86.30 under 0.2 ms CPU inference, compared with ROC AUC=87.05 for the 124 M-parameter eBERT-base teacher (Dahlmann et al., 2021). In cough classification, CoughSense (Whisper-tiny, 8.6M parameters) reaches 82.3 percent balanced accuracy and the dual-encoder model reaches 85.4 percent balanced accuracy (Vincent, 2 Jun 2026).

Domain	Encoder fine-tuning configuration	Reported outcome
20NG and MASSIVE	RoBERTa-large single-task	88.1%, 94.2%, 90.1%, 85.0%
German web data	GELECTRA-Large, URL + Content	F1 = 0.430 (best)
Astronomy papers	SciBERT_v2, 10 random chunks	0.73 Macro-F1
Federated MRI	Encoder-frozen / Full / LoRA All	0.86 AUC for all three
Cough audio	Whisper-tiny / dual-encoder fusion	82.3% / 85.4% balanced accuracy

Two recurrent debates run through this literature. The first concerns whether full end-to-end tuning is necessary. The collected results do not support a universal answer: frozen backbones, hierarchical schedules, LoRA, and shared-encoder multi-task variants can match or approach full fine-tuning in several settings (Mouheb et al., 29 Aug 2025, Heker et al., 2019, Zhu et al., 8 Oct 2025). The second concerns whether fine-tuning inherently damages pre-trained knowledge. The calibration analysis argues that, at least for absent classes, the principal failure mode is often a logit-scale bias rather than damaged features, and that a small post-hoc correction can restore performance (Mai et al., 2024). A plausible implication is that encoder-based classification fine-tuning should be evaluated not only by end-task accuracy but also by calibration, aggregation strategy, parameter-update regime, and the interaction between encoder geometry and classifier logits.