Hierarchical Fine-tuning BERT (HFT-BERT)

• HFT-BERT is a family of methods that fine-tune BERT by leveraging inherent hierarchies in labels, encoder layers, and document segments.
• It organizes the adaptation process using strategies like label graph encoding, layer-wise guidance, segment aggregation, and cyclic parameter updates.
• Empirical studies show that hierarchy-aware fine-tuning improves classification accuracy and memory efficiency across various NLP tasks.

Searching arXiv for the cited HFT-BERT-related papers to ground the article in current arXiv records. Hierarchical Fine-tuning BERT (HFT-BERT) is, as an Editor's term, a family of BERT adaptation strategies in which hierarchy determines how fine-tuning is organized. In this literature, the relevant hierarchy may be a label taxonomy, the depth of the encoder, a document’s segment structure, or the schedule by which parameter groups are updated. One paper uses the name explicitly for Chinese commodity classification, while other works instantiate closely related ideas under names such as HBGL, Layer-wise Guided Training, RoBERT, ToBERT, HiFT, and HEFT (Liu et al., 13 Aug 2025, Jiang et al., 2022, Manginas et al., 2020, Pappagari et al., 2019, Liu et al., 2024, Hill, 11 Sep 2025). The common departure from flat BERT fine-tuning is that hierarchical structure is not treated as post hoc metadata; it shapes the optimization target, the intermediate representations, or the training schedule itself.

1. Conceptual scope

The HFT-BERT literature is not defined by a single canonical architecture. Instead, it comprises several technically distinct ways of making BERT hierarchy-aware during adaptation. Some methods operate in label space, some in encoder depth, some in document segmentation, and some in the optimizer or PEFT schedule (Jiang et al., 2022, Manginas et al., 2020, Pappagari et al., 2019, Liu et al., 2024, Liu et al., 13 Aug 2025, Hill, 11 Sep 2025).

Formulation	Hierarchical object	Core mechanism
HBGL	Global label DAG and local sample-specific hierarchy	BERT encodes the label graph and predicts level-wise local hierarchies autoregressively
Layer-wise Guided Training	Hierarchy levels mapped to BERT layers	Level-specific multi-label heads supervise designated layers
RoBERT / ToBERT	Segment sequence within a long document	BERT encodes segments, then a higher-level recurrent or transformer module aggregates them
HiFT	Parameter groups across model depth	One group is updated per step, with delayed LR updates
Chinese HFT-BERT	Top-down transfer from L2 to L3 within each L1 partition	Sequential fine-tuning with separate heads
HEFT-to-BERT	Coarse-to-fine PEFT hierarchy	LoRA adaptation is followed by ReFT refinement

A central distinction across these formulations is what counts as the locus of hierarchy. In HBGL, hierarchy is the label graph and the per-document local subgraph. In Layer-wise Guided Training, hierarchy is the mapping from coarse-to-fine labels onto layer depth. In RoBERT and ToBERT, hierarchy arises from segment-to-document composition for long inputs. In HiFT and HEFT, the hierarchy is the order and granularity with which model parameters or interventions are updated (Jiang et al., 2022, Manginas et al., 2020, Pappagari et al., 2019, Liu et al., 2024, Hill, 11 Sep 2025).

2. Hierarchy in label space and prediction space

The most direct formulation of HFT-BERT appears in hierarchical text classification. HBGL defines the hierarchy as a tree or directed acyclic graph $G=(V,E)$, with $V=\{v_1,\dots,v_L\}$ the label nodes and edges encoding parent–child relations. It distinguishes two views: the global hierarchy, the full static graph shared by all documents, and the local hierarchy, a sample-specific subgraph containing only the labels relevant to a given document, organized by levels (Jiang et al., 2022). The motivation is that methods encoding only the global graph repeatedly process the same static structure, represent many labels irrelevant to the current sample, and fail to capture sample-specific structured co-occurrence.

HBGL addresses this with two tightly coupled stages. In the global stage, BERT itself is used as a graph encoder for labels. Each label embedding $\hat y_i$ is initialized by averaging the BERT token embeddings of the label’s name, then embedded as

$e_i = \hat y_i + t_1 + p_{HLevel(v_i)}.$

A global attention mask $A^{global}$ permits attention only along parent–child directions and self-attention, and masked label modeling is performed over the label graph. The score matrix is

$s = \sigma(h \hat Y^\top),$

with binary cross-entropy loss

$$L_{global} = -\sum_{i \in V_m}\sum_{j=1}^L \big[\bar y_{ij}\log s_{ij} + (1-\bar y_{ij})\log(1-s_{ij})\big].$$

During this phase, BERT is frozen and only $\hat Y$ is updated; the mask ratio is gradually increased from $0.15$ to $0.45$, with learning rates grid-searched in $V=\{v_1,\dots,v_L\}$0 and training steps in $V=\{v_1,\dots,v_L\}$1 (Jiang et al., 2022).

In the local stage, HBGL converts a document’s local hierarchy into a level-wise sequence. For level $V=\{v_1,\dots,v_L\}$2,

$V=\{v_1,\dots,v_L\}$3

BERT input concatenates document tokens, teacher-forced local hierarchy tokens $V=\{v_1,\dots,v_L\}$4, and a masked token for level $V=\{v_1,\dots,v_L\}$5. Attention masks enforce three constraints: text tokens cannot attend to label tokens, predictions at level $V=\{v_1,\dots,v_L\}$6 can attend to $V=\{v_1,\dots,v_L\}$7 but not $V=\{v_1,\dots,v_L\}$8, and masked local tokens at level $V=\{v_1,\dots,v_L\}$9 can attend to upper-level targets. The local hierarchy is factorized autoregressively,

$\hat y_i$0

and the per-level classification scores are

$\hat y_i$1

The corresponding loss is

$\hat y_i$2

Inference is iterative over levels, with $\hat y_i$3-threshold $\hat y_i$4 and $\hat y_i$5 formed by summing predicted label embeddings, or using the $\hat y_i$6 embedding if no positives are predicted (Jiang et al., 2022).

HBGL is explicitly framed as hierarchical fine-tuning because hierarchy shapes both the inputs and the losses that update BERT. Rather than fusing a separate hierarchy encoder after the text encoder, the model fine-tunes BERT under hierarchy-guided supervision. This differs from HGCLR, which uses contrastive learning and a graph encoder during training and discards the graph at test time, creating a potential train–test mismatch (Jiang et al., 2022).

A simpler label-hierarchy formulation appears in the Chinese commodity model that explicitly adopts the name HFT-BERT. There, BERT-base-Chinese is fine-tuned top-down within each $\hat y_i$7 partition: first on $\hat y_i$8, then on $\hat y_i$9, with the $e_i = \hat y_i + t_1 + p_{HLevel(v_i)}.$0-fine-tuned encoder transferred to initialize $e_i = \hat y_i + t_1 + p_{HLevel(v_i)}.$1. Each level has its own Dropout and Softmax classifier head, while the encoder is shared across levels. The per-level objective is ordinary cross-entropy,

$e_i = \hat y_i + t_1 + p_{HLevel(v_i)}.$2

and no probabilistic factorization, consistency constraint, or hierarchical regularizer is imposed. Predictions are not cascaded using parent predictions; the $e_i = \hat y_i + t_1 + p_{HLevel(v_i)}.$3 classifier does not condition explicitly on the predicted $e_i = \hat y_i + t_1 + p_{HLevel(v_i)}.$4 label (Liu et al., 13 Aug 2025). This design uses hierarchy implicitly through parameter transfer rather than through structured decoding.

3. Layer-wise supervision inside BERT

A distinct HFT-BERT line treats the encoder’s depth as the substrate for hierarchy. "Layer-wise Guided Training for BERT" does not use the term HFT-BERT, but it is an explicit instance of hierarchical fine-tuning in which specific BERT layers or layer pairs are trained to predict specific hierarchy levels (Manginas et al., 2020). The motivation is Large-Scale Multilabel Text Classification with hierarchically organized labels, where a flat head on the final layer must score labels of widely varying specificity simultaneously. The paper connects this difficulty to BERT’s over-parameterization and under-utilization.

The architecture uses a BERT-base encoder with 12 layers, hidden size 768, and 12 heads. For hierarchy levels $e_i = \hat y_i + t_1 + p_{HLevel(v_i)}.$5, several layer-to-level mappings are studied. The flat baseline uses a single head at layer 12. last-six maps $e_i = \hat y_i + t_1 + p_{HLevel(v_i)}.$6 to layers $e_i = \hat y_i + t_1 + p_{HLevel(v_i)}.$7. one-by-one maps them to layers $e_i = \hat y_i + t_1 + p_{HLevel(v_i)}.$8. in-pairs uses layer pairs $e_i = \hat y_i + t_1 + p_{HLevel(v_i)}.$9, concatenating adjacent $A^{global}$0 vectors to obtain a 1536-dimensional input. hybrid skips the first three layers, uses single layers for $A^{global}$1, $A^{global}$2, and $A^{global}$3, and paired layers for the remaining levels (Manginas et al., 2020).

Each head operates on a layer-specific pooled vector $A^{global}$4, where $A^{global}$5 for a single layer and $A^{global}$6 for a pair. The level-$A^{global}$7 classifier is

$A^{global}$8

with binary cross-entropy

$A^{global}$9

The overall objective is

$s = \sigma(h \hat Y^\top),$0

with $s = \sigma(h \hat Y^\top),$1, a weighting scheme the paper identifies as crucial (Manginas et al., 2020).

This formulation is notable for what it does not do. There is no explicit hierarchy consistency term such as a constraint that parent scores exceed child scores. Instead, label augmentation is used: if a label is assigned, all of its ancestors are added as positives. All encoder layers are fine-tuned end-to-end from the start; non-guided layers are not frozen, but they receive only indirect supervision. The reported representational analyses show larger angular distances between $s = \sigma(h \hat Y^\top),$2 vectors across layers, as well as increased attention entropy and KL divergence, indicating more diverse attention usage and better parameter utilization under structured supervision (Manginas et al., 2020).

4. Segment hierarchies for long-document classification

Another HFT-BERT formulation addresses BERT’s length limitations by making the document itself hierarchical. "Hierarchical Transformers for Long Document Classification" extends BERT to inputs longer than 512 tokens by segmenting a document into smaller chunks, encoding each chunk with BERT, and then aggregating chunk representations with a second-stage sequence model (Pappagari et al., 2019). In the paper’s terminology, the two variants are RoBERT and ToBERT.

The segmentation scheme is fixed-window and overlapping: segment length $s = \sigma(h \hat Y^\top),$3 tokens and stride $s = \sigma(h \hat Y^\top),$4. Each segment is encoded by BERT-Base, and the segment representation $s = \sigma(h \hat Y^\top),$5 is the pooled output of the final transformer block. Alternatively, segment posteriors $s = \sigma(h \hat Y^\top),$6 can be obtained by attaching a softmax classification head to BERT and fine-tuning at the segment level, assigning each segment the label of its parent document (Pappagari et al., 2019).

RoBERT aggregates the segment sequence $s = \sigma(h \hat Y^\top),$7 with a single LSTM of 100 hidden units, then applies two fully connected layers: a 30-dimensional ReLU layer and a softmax layer. ToBERT instead feeds the segment embeddings into a 2-layer transformer, optionally with learned segment-level positional embeddings. In both cases, BERT fine-tuning is performed independently of the top aggregator; the model is not jointly fine-tuned end-to-end with the aggregator in this paper (Pappagari et al., 2019).

The hierarchical structure is therefore segment-to-document rather than taxonomic. The recurrent variant preserves chronological order and is naturally streaming, because the final decision is made after the last segment has been consumed. The transformer variant is not described as streaming, since standard self-attention requires access to the full segment sequence. The paper characterizes RoBERT as reducing complexity to $s = \sigma(h \hat Y^\top),$8 when segment size $s = \sigma(h \hat Y^\top),$9 is fixed, while ToBERT has document-level self-attention cost $$L_{global} = -\sum_{i \in V_m}\sum_{j=1}^L \big[\bar y_{ij}\log s_{ij} + (1-\bar y_{ij})\log(1-s_{ij})\big].$$0, though in practice the number of segments $$L_{global} = -\sum_{i \in V_m}\sum_{j=1}^L \big[\bar y_{ij}\log s_{ij} + (1-\bar y_{ij})\log(1-s_{ij})\big].$$1 remains much smaller than the original token length $$L_{global} = -\sum_{i \in V_m}\sum_{j=1}^L \big[\bar y_{ij}\log s_{ij} + (1-\bar y_{ij})\log(1-s_{ij})\big].$$2 (Pappagari et al., 2019).

This formulation broadens the meaning of HFT-BERT. The hierarchy is not in the labels or in model depth, but in the composition of local segment representations into a document representation. It is a hierarchical fine-tuning strategy because BERT is fine-tuned at the segment level and then embedded into a higher-level document classifier.

5. Hierarchical optimization and parameter-update schedules

A further usage of HFT-BERT concerns how BERT is optimized rather than how labels are encoded. HiFT, described as a hierarchical full-parameter fine-tuning strategy, fine-tunes all parameters over the course of training but not all at once. Layers are divided into groups, embeddings and head are treated as single layers, and one group is updated per step while the others are frozen (Liu et al., 2024). The hierarchy is therefore the ordered partition of the model.

HiFT maintains a queue $$L_{global} = -\sum_{i \in V_m}\sum_{j=1}^L \big[\bar y_{ij}\log s_{ij} + (1-\bar y_{ij})\log(1-s_{ij})\big].$$3 of group identifiers and visits groups cyclically under one of three strategies: bottom2up, top2down, or random. The learning-rate schedule is delayed so that the LR is updated only after all groups have been updated once. This design is intended to mitigate instability from uneven update amplitudes across groups and preserve convergence (Liu et al., 2024).

The memory analysis is explicit. For AdamW in 32-bit precision, full-parameter fine-tuning requires

$$L_{global} = -\sum_{i \in V_m}\sum_{j=1}^L \big[\bar y_{ij}\log s_{ij} + (1-\bar y_{ij})\log(1-s_{ij})\big].$$4

At HiFT step $$L_{global} = -\sum_{i \in V_m}\sum_{j=1}^L \big[\bar y_{ij}\log s_{ij} + (1-\bar y_{ij})\log(1-s_{ij})\big].$$5, if only subset $$L_{global} = -\sum_{i \in V_m}\sum_{j=1}^L \big[\bar y_{ij}\log s_{ij} + (1-\bar y_{ij})\log(1-s_{ij})\big].$$6 is updated,

$$L_{global} = -\sum_{i \in V_m}\sum_{j=1}^L \big[\bar y_{ij}\log s_{ij} + (1-\bar y_{ij})\log(1-s_{ij})\big].$$7

Under even partitioning into $$L_{global} = -\sum_{i \in V_m}\sum_{j=1}^L \big[\bar y_{ij}\log s_{ij} + (1-\bar y_{ij})\log(1-s_{ij})\big].$$8 groups, the reduction ratio is

$$L_{global} = -\sum_{i \in V_m}\sum_{j=1}^L \big[\bar y_{ij}\log s_{ij} + (1-\bar y_{ij})\log(1-s_{ij})\big].$$9

The paper reports that HiFT reduces the number of trainable parameters per step by about $\hat Y$0 on average compared to FPFT, can save more than $\hat Y$1 GPU memory compared with standard full-parameter fine-tuning for a 7B model, and enables full-parameter fine-tuning of a 7B model on single 48G A6000 with precision 32 using AdamW, without using memory-saving techniques (Liu et al., 2024).

In a PEFT setting, HEFT introduces a coarse-to-fine hierarchy: first LoRA, then ReFT. The abstract reports results on Llama-2-7B and BoolQ, but the detailed presentation includes a concrete BERT adaptation recipe (Hill, 11 Sep 2025). LoRA is the coarse stage, with

$\hat Y$2

where $\hat Y$3 and $\hat Y$4 are low-rank trainable matrices and $\hat Y$5 is frozen during the LoRA phase. ReFT is the fine stage, adding a residual intervention to hidden states,

$\hat Y$6

and in the LoReFT instantiation,

$\hat Y$7

For BERT-base, the recommended LoRA placement is on query and value projections across all layers, optionally extending to $\hat Y$8 and $\hat Y$9 in the FFN. With rank $0.15$0, Q+V-only LoRA introduces approximately 294,912 trainable parameters, while Q+V+$0.15$1+$0.15$2 introduces approximately 1.11M. A single LoReFT at one layer with $0.15$3 and $0.15$4 adds approximately 3,088 parameters (Hill, 11 Sep 2025).

These optimizer-level methods expand HFT-BERT beyond hierarchical classification. Here, hierarchical fine-tuning means that the update path through parameter space is itself structured: either by cycling through groups of layers, as in HiFT, or by composing a broad weight-space adaptation with a precise representation-space intervention, as in HEFT (Liu et al., 2024, Hill, 11 Sep 2025).

6. Empirical behavior across tasks

The empirical literature shows that hierarchy-aware fine-tuning can improve performance, but the magnitude of improvement depends strongly on the task, hierarchy depth, and formulation.

In hierarchical text classification, HBGL reports gains over HGCLR on all three benchmarks. On WOS, HBGL obtains $0.15$5 Micro/Macro-F1 versus $0.15$6 for HGCLR. On NYT, it reports $0.15$7 versus $0.15$8. On RCV1-V2, it reports $0.15$9 versus $0.45$0. The reported gains over HGCLR are $0.45$1 on WOS, $0.45$2 on NYT, and $0.45$3 on RCV1-V2, with the paper noting that HBGL particularly shines on deeper hierarchies such as NYT, where $0.45$4 (Jiang et al., 2022). Ablations further report that global hierarchy-aware label embeddings learned with BERT outperform random initialization, label-name averages, and a GAT encoder, and that local hierarchy-aware fine-tuning outperforms both flat multi-label fine-tuning and sequence-to-sequence fine-tuning (Jiang et al., 2022).

For layer-wise guided training, the strongest variant is last-six. On EURLEX57K with BERT-base, the flat baseline reports micro $0.45$5 and macro $0.45$6, while last-six reaches micro $0.45$7 and macro $0.45$8. On MIMIC-III with SciBERT, flat reports micro $0.45$9 and macro $V=\{v_1,\dots,v_L\}$00, while last-six reaches micro $V=\{v_1,\dots,v_L\}$01 and macro $V=\{v_1,\dots,v_L\}$02. The largest per-level gains are typically at deeper levels, such as $V=\{v_1,\dots,v_L\}$03 (Manginas et al., 2020).

For long-document classification, the effect of hierarchical aggregation is clearest on the longest, multi-topic data. Using segment representations $V=\{v_1,\dots,v_L\}$04 from a fine-tuned BERT, RoBERT and ToBERT report accuracies of $V=\{v_1,\dots,v_L\}$05 and $V=\{v_1,\dots,v_L\}$06 on CSAT, $V=\{v_1,\dots,v_L\}$07 and $V=\{v_1,\dots,v_L\}$08 on 20 Newsgroups, and $V=\{v_1,\dots,v_L\}$09 and $V=\{v_1,\dots,v_L\}$10 on Fisher. With frozen segment features, ToBERT also strongly outperforms RoBERT on Fisher, reporting $V=\{v_1,\dots,v_L\}$11 versus $V=\{v_1,\dots,v_L\}$12. The paper further reports that both extensions are quick to fine-tune and can converge after as little as 1 epoch on small, domain-specific data (Pappagari et al., 2019).

The explicitly named Chinese HFT-BERT model shows its strongest advantage at $V=\{v_1,\dots,v_L\}$13. At $V=\{v_1,\dots,v_L\}$14, it is best only on books, where it reaches $V=\{v_1,\dots,v_L\}$15, while Flat-CNN is stronger on fresh, household appliances, and digital products. At $V=\{v_1,\dots,v_L\}$16, however, HFT-BERT reports $V=\{v_1,\dots,v_L\}$17 on fresh, $V=\{v_1,\dots,v_L\}$18 on household appliances, $V=\{v_1,\dots,v_L\}$19 on digital products, and $V=\{v_1,\dots,v_L\}$20 on books, outperforming Flat-CNN, Hier-CNN, and HFT-CNN on all four $V=\{v_1,\dots,v_L\}$21 partitions. The largest gap is on books, where the CNN baselines remain around $V=\{v_1,\dots,v_L\}$22 to $V=\{v_1,\dots,v_L\}$23 (Liu et al., 13 Aug 2025).

The optimization-oriented papers report a different kind of empirical outcome. HiFT emphasizes memory and convergence properties rather than hierarchical classification metrics, reporting memory savings of approximately $V=\{v_1,\dots,v_L\}$24 to $V=\{v_1,\dots,v_L\}$25 on RoBERTa-base and RoBERTa-large, approximately $V=\{v_1,\dots,v_L\}$26 to $V=\{v_1,\dots,v_L\}$27 on GPT-Neo $V=\{v_1,\dots,v_L\}$28B, and approximately $V=\{v_1,\dots,v_L\}$29 to $V=\{v_1,\dots,v_L\}$30 on LLaMA-2 $V=\{v_1,\dots,v_L\}$31B, with performance competitive with FPFT and PEFT across tasks (Liu et al., 2024). HEFT reports on BoolQ that a model trained for only three LoRA epochs plus three ReFT epochs reaches $V=\{v_1,\dots,v_L\}$32 validation accuracy, exceeding LoRA-only at 20 epochs ($V=\{v_1,\dots,v_L\}$33) and ReFT-only at 20 epochs ($V=\{v_1,\dots,v_L\}$34); a 20+20 schedule reaches $V=\{v_1,\dots,v_L\}$35 (Hill, 11 Sep 2025).

7. Limitations, misconceptions, and open directions

A recurrent misconception is that HFT-BERT denotes a single standardized model. The literature does not support that reading. The name is explicit in the Chinese commodity paper, but several other works instantiate equivalent or closely related ideas under different names and with different notions of hierarchy (Liu et al., 13 Aug 2025, Jiang et al., 2022, Manginas et al., 2020, Pappagari et al., 2019, Liu et al., 2024, Hill, 11 Sep 2025).

A second misconception is that hierarchical fine-tuning necessarily enforces hierarchy consistency in the loss. That is not generally true. Layer-wise Guided Training uses no explicit parent–child consistency regularization, relying instead on label augmentation (Manginas et al., 2020). The Chinese HFT-BERT model also uses no explicit probabilistic factorization or consistency constraint, and its $V=\{v_1,\dots,v_L\}$36 classifier does not condition on predicted $V=\{v_1,\dots,v_L\}$37 labels (Liu et al., 13 Aug 2025). HBGL is more structurally constrained, but its hierarchy awareness comes from masks, shared label embeddings, and autoregressive decoding rather than from a standalone monotonicity penalty (Jiang et al., 2022).

Each formulation also has distinct failure modes. HBGL incurs iterative level-wise decoding overhead, depends on accurate early-level predictions, can suffer error cascades, and may inject suboptimal structure if the taxonomy is noisy or poorly aligned to semantics. Its per-level summation $V=\{v_1,\dots,v_L\}$38 may under-model complex DAG dependencies, and teacher forcing during training can widen the train–test gap if upper levels are difficult (Jiang et al., 2022). RoBERT and ToBERT introduce segment-level label noise because document labels are assigned to each segment during segment-level fine-tuning, and ToBERT reintroduces $V=\{v_1,\dots,v_L\}$39 complexity at the segment level (Pappagari et al., 2019). HiFT can require $V=\{v_1,\dots,v_L\}$40–$V=\{v_1,\dots,v_L\}$41, and up to $V=\{v_1,\dots,v_L\}$42, more epochs at the same batch size; it also introduces overhead from toggling requires_grad and moving optimizer states between CPU and GPU (Liu et al., 2024). HEFT leaves the reverse order ReFT$V=\{v_1,\dots,v_L\}$43LoRA and simultaneous training unevaluated, so order dependence remains open (Hill, 11 Sep 2025). The Chinese commodity model assumes that $V=\{v_1,\dots,v_L\}$44 is known or handled externally, since separate models are trained per $V=\{v_1,\dots,v_L\}$45 partition (Liu et al., 13 Aug 2025).

The open directions identified in the literature are correspondingly heterogeneous. For taxonomy-aware HTC, potential extensions include explicit hierarchical consistency regularizers, focal loss for class imbalance, and level-dependent thresholds (Jiang et al., 2022). For layer-wise guided training, proposed directions include applying the same mapping principle to RoBERTa, DeBERTa, or domain-specific encoders, as well as guiding attention heads to hierarchy parts (Manginas et al., 2020). For long documents, end-to-end joint fine-tuning of BERT and the aggregator remains an obvious unresolved step (Pappagari et al., 2019). For optimizer- and PEFT-level HFT-BERT, future work includes finer grouping granularity, automated layer or rank selection, adaptive ReFT placement, and multi-skill reuse of a shared LoRA-adapted foundation (Liu et al., 2024, Hill, 11 Sep 2025).

Taken together, these lines of work show that HFT-BERT is best understood not as a single model family with one invariant architecture, but as a research program: BERT is fine-tuned under an explicitly hierarchical inductive bias, and the hierarchy may reside in labels, layers, segments, or update schedules.