---
title: Encoder-Based Classification Fine-Tuning
url: https://www.emergentmind.com/topics/encoder-based-classification-fine-tuning
type: topic
---

# Encoder-Based Classification Fine-Tuning

Encoder-based classification fine-tuning is the adaptation of a pre-trained encoder to a supervised classification objective by attaching a task-specific head to a pooled representation and then updating some or all model parameters on downstream data. In BERT-style sequence classification, a common practice is to feed the embedding of the final-layer `[CLS]` token to a classification layer and fine tune the parameters of the encoder and classifier jointly [1911.03918]. Across the literature, this pattern appears in encoder-only NLP models such as BERT, RoBERTa, SciBERT, CamemBERT, mBERT, XLM-R, and BERTurk, and also in non-text encoders in medical imaging, audio, and multimodal image-text systems, where pooled latent features are mapped to class logits by linear or lightweight adapter heads [1905.05583], [2412.08587], [2508.21458].

## 1. Canonical architectural pattern

The most common formulation uses a pre-trained encoder to map an input sequence to contextual hidden states and then applies a shallow classification head to a pooled representation. For BERT fine-tuning, the input is tokenized with WordPiece, prefixed by `[CLS]`, suffixed or segment-split by `[SEP]`, and processed by 12 Transformer blocks; the final hidden state at the `[CLS]` position, $h \in \mathbb{R}^{768}$, is passed through a single linear layer and softmax, $p(c \mid h)=\mathrm{softmax}(Wh+b)$ [1905.05583]. RoBERTa fine-tuning in single-task text classification follows the same pattern, with a sequence head $W_{\mathrm{seq}} \in \mathbb{R}^{K \times D}$ and bias $b_{\mathrm{seq}}$, $\hat y_i=\mathrm{softmax}(W_{\mathrm{seq}} h_{(\mathrm{CLS})}+b_{\mathrm{seq}})$, while token classification uses per-token hidden vectors and a separate token head [2412.08587].

Binary and multi-label variants retain the encoder and replace the final activation. In German web topic detection, a single feed-forward layer is appended to the `[CLS]` embedding, with architecture `Dropout(p = 0.1) → Linear(H→1) → Sigmoid`, producing a probability for the positive class [2407.16516]. In French plant-health bulletin classification, CamemBERT-base and multilingual BERT use the final hidden state of `[CLS]`, a fully connected layer $W\in\mathbb{R}^{H\times 2}$, and an element-wise sigmoid to produce independent probabilities for the `pest` and `disease` labels [2102.00838]. BERTurk text classification uses `[CLS]`, dropout $(p=0.1)$, a dense layer, and softmax over $C$ logits [2401.17396].

Multi-task encoder fine-tuning extends the same design by attaching several heads to one shared encoder. In astronomy knowledge extraction, SciBERT is fine-tuned with one multiclass telescope head and four independent binary heads, all fed by the shared final `[CLS]` representation $h_{\text{CLS}}\in\mathbb{R}^{H}$ with $H=768$ [2511.08204]. In RoBERTa-based MASSIVE experiments, intent detection uses a sequence head over `[CLS]`, while slot filling uses a token head over per-token hidden states, with all RoBERTa layers shared [2412.08587].

Outside text, the architectural template persists but the pooling operator changes. In federated MRI-based dementia classification, the SAM-Med3D encoder produces a latent tensor $z=g_\theta(x)\in\mathbb{R}^{384\times 8\times 8\times 8}$; a linear head performs global average pooling to $\bar z\in\mathbb{R}^{384}$ and then applies a single linear layer plus softmax, while larger alternatives add small 3D convolutional adapters before the classifier [2508.21458]. In cough classification, Whisper-tiny’s decoder is discarded, the encoder output is pooled by active-frame QKV attention over the first $K=200$ tokens, and the pooled representation is sent to the disease head [2606.02998].

## 2. Loss functions and prediction regimes

The dominant supervised objective is cross-entropy or binary cross-entropy, with straightforward extensions to token labeling, multi-task learning, and multi-label prediction. For single-task sequence classification in RoBERTa, the loss is
$$
L_{\mathrm{CE}}(\theta)=-\sum_{i=1}^{N}\sum_{k=1}^{K}\mathbf{1}\{y_i=k\}\cdot \log p_\theta(y_i=k\mid x_i),
$$
and slot filling uses token-wise cross-entropy,
$$
L_{\mathrm{slot}}(\theta)= - \sum_{i=1}^N \sum_{j=1}^{L_i}\sum_{s=1}^{S}\mathbf{1}\{y_{ij}=s\}\cdot \log p_\theta(y_{ij}=s\mid x_i).
$$
In multi-task settings, the cited work simply sums the losses, $L_{\mathrm{joint}}(\theta)=L_{\mathrm{intent}}(\theta)+L_{\mathrm{slot}}(\theta)$ [2412.08587].

Binary classification tasks use the standard binary cross-entropy. In German webpage topic detection,
$$
L(\theta)= -\frac{1}{N}\sum_{i=1}^{N}\bigl[y_i\log \hat y_i+(1-y_i)\log(1-\hat y_i)\bigr],
$$
with early stopping on validation F1 and regularization through dropout and weight decay [2407.16516]. In French plant-health bulletin classification, the objective sums binary cross-entropy over the two labels, and the encoder is fine-tuned end-to-end with Adam [2102.00838]. Astronomy knowledge extraction combines a multiclass cross-entropy for telescope prediction with four binary cross-entropy losses for the Boolean labels, with default equal weights $\lambda_{\rm tel}=1$ and $\lambda_{\rm bool}^{(i)}=1$ [2511.08204].

Auxiliary objectives are used when plain supervised loss is considered insufficient. In lightweight transformer pre-finetuning for text classification and NER, the text-classification branch uses an InfoNCE contrastive loss with temperature $\tau=0.05$, the NER branch uses cross-entropy, and the multi-task pre-finetuning objective is $L_{\mathrm{total}}=L_{\mathrm{CL}}+L_{\mathrm{NER}}$ [2510.07566]. In cough classification, the total batch loss is
$$
L_{\mathrm{total}}=L_{\mathrm{focal}}+0.3\,\lambda(t)\,L_{\mathrm{dom}}+0.1\,L_{\mathrm{SupCon}},
$$
combining focal loss, a domain-adversarial branch with gradient reversal, and supervised contrastive loss [2606.02998]. In zero-shot X-ray pathology classification, contrastive image-text encoders are fine-tuned with a modified CLIP-style InfoNCE objective, augmented by positive-pair loss relaxation and random sentence sampling [2212.07050].

These formulations show that encoder-based classification fine-tuning is not restricted to single-label softmax classification. The same encoder can support binary, multi-class, multi-label, token-level, contrastive, and hybrid objectives, provided that the head and loss are matched to the prediction regime.

## 3. Parameter-update strategies and optimization control

The cited work describes a wide range of update regimes: full end-to-end fine-tuning, linear probing with a frozen encoder, parameter-efficient tuning, staged unfreezing, and hierarchical freezing. Classical BERT fine-tuning updates all encoder layers plus the classifier, but several studies report that careful control of learning rates and parameter movement is important. In the exhaustive BERT text-classification study, aggressive learning rates $(\ge 1\times 10^{-4})$ cause rapid degradation of pre-trained features, whereas a base learning rate of $2\times 10^{-5}$ with warm-up is low enough to preserve knowledge yet flexible enough to adapt; layer-wise learning-rate decay with $\xi \approx 0.95$ produced the best IMDb error among the reported settings [1905.05583].

Optimization can also be improved by intervening on the pooled representation. A systematic analysis of BERT sequence classification reported a biased embedding distribution issue before fine tuning: embedding values of `[CLS]` concentrate on a few dimensions and are non-zero centered, which brings challenge to the optimization process during fine-tuning because gradients of `[CLS]` embedding may explode and result in degraded model performance. The paper proposes several simple normalization methods to modify the `[CLS]` embedding during fine-tuning, and the normalized embedding shows improvements on several text classification tasks [1911.03918].

Selective tuning is often motivated by efficiency or data scarcity. In federated MRI-based dementia classification, full fine-tuning, encoder-frozen fine-tuning, and LoRA all achieved $0.86$ AUC on the pooled test set with overlapping confidence intervals, while freezing the backbone reduced communication by approximately $25\times$ and GPU memory by approximately $3\times$ [2508.21458]. In hierarchical fine-tuning for liver lesion classification, transferring encoder-decoder weights from LiTS and then progressively freezing or unfreezing encoder blocks improved both Dice and classification accuracy relative to training from scratch, with hierarchical unfreezing reaching accuracy $0.73$ in 3-fold cross-validation [1907.13409]. In CoughSense, training is explicitly two-phase: the entire Whisper encoder is frozen for epochs 1–3 and then unfrozen for epochs 4–25 with differential learning rates, $\eta_{\mathrm{enc}}=2\times 10^{-5}$ and $\eta_{\mathrm{head}}=1\times 10^{-3}$ [2606.02998].

Parameter-efficient tuning introduces modularity rather than simply reducing trainable parameter count. In multi-task pre-finetuning of lightweight encoders, MTPF-TPL attaches two LoRA modules to the last $L$ transformer layers: `LoRA_TC` is updated only with the contrastive text-classification loss, `LoRA_NER` only with the NER loss, while the shared backbone $W$ is updated with both losses [2510.07566]. This design directly addresses the reported problem that naïve multi-task pre-finetuning introduces conflicting optimization signals that degrade overall performance [2510.07566].

A further qualification concerns what is often described as catastrophic forgetting. Fine-tuning on a subset of classes can drastically degrade accuracy on absent classes, but a detailed analysis argues that the fine-tuned model often produces more discriminative features for those absent classes; the main problem is discrepant logit scales between fine-tuning classes and absent classes. An additive post-processing calibration,
$$
\hat y=\arg\max_c\,[z_c^T(x)+\gamma \cdot \mathbf{1}[c\in U]],
$$
recovers and often exceeds pre-trained performance on absent classes [2409.16223]. This suggests that poor post-fine-tuning behavior is not always evidence that encoder features have been destroyed.

## 4. Input construction, truncation, pooling, and aggregation

A central engineering problem in encoder-based classification fine-tuning is the mismatch between input length and encoder context window. The basic BERT recipe uses a maximum length of 512 sub-word tokens [1905.05583]. For long documents, truncation strategy matters: on IMDb and Sogou, keeping the head and tail of the document outperformed head-only, tail-only, or hierarchical pooling, with reported error rates of `5.42 / 2.43` for head+tail versus `5.63 / 2.58` for head-only and `5.44 / 3.17` for tail-only [1905.05583].

When truncation is too lossy, document chunking and aggregation are used. In astronomy papers that exceed SciBERT’s 512-token limit, each document is split into non-overlapping 512-token chunks; during training, up to $m=10$ chunks per document are sampled uniformly at random, and at test time all segments are processed and aggregated by majority voting [2511.08204]. The reported Macro-F1 improves from `0.72` for `SciBERT_v1 (first 512 tokens only)` to `0.73` for `SciBERT_v2 (10 random chunks)`, and the discussion attributes the gain to varied parts of the paper, particularly acknowledgments and grant sections [2511.08204].

German web topic classification uses a related but distinct strategy. Content is split into overlapping chunks with `384 tokens max, 64-token overlap`, and each chunk inherits the URL label; the framework also compares `URL only`, `Content only`, and `URL + Content` feature constructions [2407.16516]. In chest X-ray report fine-tuning, random sentence sampling selects $n=3$ sentences from the report in each training iteration to create alternative positive text views without introducing explicit pathology labels [2212.07050].

Signal dilution can arise even when the formal context window is sufficient. CoughSense observes that a 3-second cough yields only approximately 150 non-zero tokens out of Whisper’s 1500-token, 30-second input window. It therefore restricts attention pooling to the first $K=200$ tokens and replaces mean pooling with learned QKV attention pooling. In the ablation, mean-pooling all 1500 tokens gives `73.1 %` balanced accuracy, active-frame cropping raises it to `78.2 %`, and adding QKV attention pooling raises it to `80.4 %` [2606.02998]. This suggests that input aggregation is often as consequential as the encoder backbone itself.

## 5. Pre-finetuning, continued pre-training, and multi-task organization

Encoder-based classification fine-tuning is frequently preceded by additional unsupervised or weakly supervised adaptation. In BERT text classification, continued masked-LM and next-sentence pre-training on unlabeled data can be performed within-task, in-domain, or cross-domain; in-domain pre-training produced the largest average error reduction over vanilla BERT fine-tuning, approximately `18.6%`, compared with approximately `16%` for within-task and approximately `14.4%` for cross-domain pre-training [1905.05583]. In plant-health bulletin classification, the supervised fine-tuning stage is preceded by domain-adaptive pre-training on unlabeled XML BSV paragraphs for `2 epochs`, with batch size `8`, initial learning rate `1×10⁻⁴`, and optimizer `Adam` [2102.00838]. In production query-title relevance classification, domain-specific pre-training of eBERT on BooksCorpus, Wikipedia, and `1 B item titles` improves the downstream teacher from `ROC AUC=82.26 %` for off-the-shelf BERT-base to `ROC AUC=87.05` for eBERT-base [2108.10197].

Pre-finetuning can also be organized as a bridge between task families. MTPF-TPL jointly pre-finetunes a single lightweight encoder for a text-classification contrastive objective and a pseudo-labeled NER objective using task-primary LoRA modules, and then uses the shared encoder plus classification adapter for downstream text classification [2510.07566]. The reported average accuracy rises from `55.1%` for base MiniLM to `63.9%` for pre-finetuning on text classification only and `64.1%` for MTPF-TPL [2510.07566].

Multi-task supervised fine-tuning typically shares the encoder and keeps heads separate. RoBERTa fine-tuning on MASSIVE uses one shared encoder with one sequence head for intent and one token head for slot filling, optimized by the unweighted sum $L_{\mathrm{joint}}=L_{\mathrm{intent}}+L_{\mathrm{slot}}$ [2412.08587]. Astronomy knowledge extraction similarly shares all SciBERT layers across one multiclass head and four Boolean heads, again with equal weighting [2511.08204]. The advantage is consolidation: one encoder handles related classification tasks in a single pass.

The supervision source need not be manual labels. For monolingual and zero-shot polylingual neural topic modeling, one strategy is to run LDA on the target corpus, label each document by its most probable LDA topic, and fine-tune the encoder on that topic classification task using cross-entropy [2104.05064]. The same work also integrates the topic classification objective directly into the contextualized topic model, with
$$
L_{\mathrm{total}}=L_{\mathrm{ELBO}}+\lambda L_{\mathrm{cls}}.
$$
The reported findings distinguish objectives by downstream effect: in-domain topic classification gives the highest monolingual NPMI, while NLI fine-tuning gives the best cross-lingual transfer [2104.05064].

## 6. Empirical behavior, trade-offs, and recurrent debates

Reported outcomes vary by task, modality, and evaluation protocol, but several recurring patterns are visible. Encoder-only fine-tuning remains a strong and inexpensive baseline, yet it is not uniformly dominant against larger decoder LLMs. On 20 Newsgroups and MASSIVE, single-task `RoBERTa-large` reaches `88.1%` on 20NG twenty-class, `94.2%` on 20NG seven-class, `90.1%` on MASSIVE intent, and `85.0%` token-level F1 on MASSIVE slot filling, while fully fine-tuned `Llama3-70B` is reported to outperform `RoBERTa-large` by approximately `4–5 percentage points`; at the same time, `RoBERTa large still a strong baseline when compute or data for LLM fine-tuning is limited` [2412.08587].

Under noisy, unbalanced, and domain-specific conditions, encoder fine-tuning often outperforms alternatives that look competitive on cleaner evaluations. In German policy-topic detection, `GELECTRA-Large` with `URL + Content` reaches `F1 = 0.430 (best)` on the complete test, the average `URL only` result is `0.037`, and `URL + Content` gives `average F1=0.308`, a `+ 40.8% rel. gain` over URL-only. The same study states that fine-tuned encoders consistently outperform in-context learning under realistic unbalanced and noisy conditions, even though few-shot Vicuna performs strongly on the balanced test [2407.16516].

Domain specialization repeatedly matters. In French plant-health bulletins, multilingual BERT slightly outperforms CamemBERT with `Weighted F₁=0.809` and `ROC AUC≈0.911` [2102.00838]. In astronomy knowledge extraction, `SciBERT_v2 (10 random chunks)` reaches `0.73` Macro-F1 on the held-out test set, compared with `0.31` for `OpenAI-GPT-OSS-20B (zero/few-shot)` [2511.08204]. In chest X-ray zero-shot pathology classification, the proposed fine-tuning strategy improves overall zero-shot pathology classification across four datasets and three pre-trained models, with an `average macro AUROC increase of 4.3%` reported in the abstract [2212.07050].

Efficiency and deployment constraints frequently favor partial tuning or encoder reuse. In federated MRI-based dementia classification, `Encoder-frozen`, `Full fine-tuning`, and `LoRA All` all report `0.86` AUC, while encoder freezing reduces communication overhead and memory usage [2508.21458]. In production query-title relevance classification, eBERT fine-tuning provides the teacher, and a distilled `BertBiLSTM` student achieves `ROC AUC=86.30` under `0.2 ms` CPU inference, compared with `ROC AUC=87.05` for the `124 M`-parameter eBERT-base teacher [2108.10197]. In cough classification, `CoughSense (Whisper-tiny, 8.6M parameters)` reaches `82.3 percent balanced accuracy` and the dual-encoder model reaches `85.4 percent balanced accuracy` [2606.02998].

| Domain | Encoder fine-tuning configuration | Reported outcome |
|---|---|---|
| 20NG and MASSIVE | RoBERTa-large single-task | `88.1%`, `94.2%`, `90.1%`, `85.0%` |
| German web data | GELECTRA-Large, URL + Content | `F1 = 0.430 (best)` |
| Astronomy papers | SciBERT_v2, 10 random chunks | `0.73` Macro-F1 |
| Federated MRI | Encoder-frozen / Full / LoRA All | `0.86` AUC for all three |
| Cough audio | Whisper-tiny / dual-encoder fusion | `82.3%` / `85.4%` balanced accuracy |

Two recurrent debates run through this literature. The first concerns whether full end-to-end tuning is necessary. The collected results do not support a universal answer: frozen backbones, hierarchical schedules, LoRA, and shared-encoder multi-task variants can match or approach full fine-tuning in several settings [2508.21458], [1907.13409], [2510.07566]. The second concerns whether fine-tuning inherently damages pre-trained knowledge. The calibration analysis argues that, at least for absent classes, the principal failure mode is often a logit-scale bias rather than damaged features, and that a small post-hoc correction can restore performance [2409.16223]. A plausible implication is that encoder-based classification fine-tuning should be evaluated not only by end-task accuracy but also by calibration, aggregation strategy, parameter-update regime, and the interaction between encoder geometry and classifier logits.

Source: https://www.emergentmind.com/topics/encoder-based-classification-fine-tuning