Hierarchical Character Language Model

• HCLM is a hierarchical model that integrates fine-grained character processing with higher-order word-level context for comprehensive language modeling.
• Its design employs mechanisms like reset and context exchange between levels to efficiently bridge short-range dependencies and long-range semantics.
• HCLMs have demonstrated competitive or superior performance with enhanced robustness to input corruption, domain shifts, and open-vocabulary challenges.

A Hierarchical Character LLM (HCLM) is a neural architecture for language modeling that incorporates explicit multi-level modeling of character and higher-order linguistic units (typically words), while retaining a fully open-vocabulary and character-aware approach at its interface. HCLMs have been instantiated both with recurrent stacking (using LSTMs) and with hierarchical Transformer-based modules, unifying short-range character-level dependencies with longer-range word- or sequence-level context. These models have demonstrated competitive or superior performance to traditional word- or subword-tokenized architectures, and exhibit distinct robustness to input corruption, domain shift, and vocabulary-open scenarios (Hwang et al., 2016, Neitemeier et al., 17 Jan 2025, Sun et al., 2023).

1. Model Architectures

1.1 Recurrent HCLMs

The foundational Hierarchical Character-Level LLM architecture (Hwang et al., 2016) consists of a two-level recurrent stack:

• Level-1 (Character Module): A multi-layer LSTM operates at every character timestep, receiving a 1-of-|C| vector $x_t$ at time $t$, updating a latent state $h_{1,t}$, and generating a softmax over the next character in the sequence. This module runs synchronously on the “character clock” ($c_{1,t}\equiv1$).
• Level-2 (Word Module): A parallel LSTM is activated only at word (or sentence) boundaries, marked by special tokens (“<w>”, “<s>”). Its state $h_{2,t}$ updates only when $x_t$ is a boundary token; it otherwise remains static.

A key innovation is the reset and context exchange between levels: upon each word boundary (i.e., when $c_{2,t}=1$), the word-level module emits a fixed-dimensional context vector $v_{2,t}$ to inform the character module at the start of the subsequent word. Simultaneously, the character module’s state is reset ($r_{1,t}=c_{2,t}$), and an embedding summarizing the prior word’s character sequence ($v_{1,t}$) is sent upwards to the word-level module. Input and output operations remain character-level only.

1.2 Transformer HCLMs

Recent HCLMs are based on hierarchical Transformer stacks (Neitemeier et al., 17 Jan 2025, Sun et al., 2023):

• Character-Level Encoder: Each textual segment (typically a word) is represented as a sequence of characters prefixed by a special token (e.g., $[\mathrm{WORD\_CLS}]$ or [W]). A shallow bidirectional (or unidirectional, depending on task) Transformer stack produces a character-aware word embedding by reading each sequence.
• Word-Level Backbone: The sequence of word embeddings is processed by a deep inter-word Transformer (e.g., 12–36 layers depending on scale) which forms contextualized representations for each word in the sentence/document.
• Character-Level Decoder: For generative/autoregressive settings, the word-level context is projected back and, together with the character-level embeddings, used by a compact Transformer decoder to model or predict subsequent characters/bytes.

This arrangement decouples modeling granularity: intra-word dependencies are handled by lightweight per-word character Transformers and inter-word semantics by deep sequence-level modules. Input/output remain fully open-vocabulary via byte/character-level softmax layers.

2. Mathematical Formalism

The general structure across both recurrent and Transformer HCLMs is as follows:

• Generic RNN Update with Clock and Reset (Hwang et al., 2016):

$$s_t = (1-c_t)(1-r_t)\cdot s_{t-1} + c_t f(x_t, (1-r_t) s_{t-1}) \ y_t = g(s_t)$$

Here, $c_t \in \{0,1\}$ is the module’s external clock (activates only at zoomed-in levels), $r_t\in\{0,1\}$ is the reset signal, $f$ is the transition (e.g., LSTM cell), $g$ the output head (softmax for character prediction).

• Transformer HCLM Encoding (Sun et al., 2023, Neitemeier et al., 17 Jan 2025):
  • For word $i$ with $C_i$ characters, $x_{i,0}$ is a [WORD_CLS] token, $x_{i,1}, ..., x_{i,C_i}$ are characters. Embedded as $\mathbf{e}_{i,j} = E[x_{i,j}]$ and passed (with position encodings) through $L_\mathrm{intra}$ self-attention layers to yield intra-word hidden states. The word embedding is the [CLS] hidden at the terminal layer: $r_i = \mathbf{h}_{i,0}^{(L_\mathrm{intra})}$.
  • The sequence $r_1, ..., r_N$ (for $N$ words) with position encodings forms the input to a deep sequence-level Transformer.
• Prediction Objectives:
  • Autoregressive HCLM: Byte-level or character-level softmaxes over next unit, conditioned hierarchically according to prior context (Neitemeier et al., 17 Jan 2025).
  • Masked Language Modeling HCLM: Whole-word masked character prediction, where entire words are masked and characters are recovered via in-word decoding (Sun et al., 2023).

3. Training Procedure and Implementation Details

The training regime is determined by the architectural choices and application:

• Recurrent HCLM (Hwang et al., 2016): Trained via character-level cross-entropy loss, optimized by truncated BPTT with ADADELTA (Nesterov momentum), gradient clipping through ADADELTA’s adaptive step-size. No explicit regularization or dropout. Both clocks and resets are fully differentiable for end-to-end training. HLSTM-B instantiation uses 4 layers per module with 512 or 1024 LSTM cells.
• Transformer HCLMs (Sun et al., 2023, Neitemeier et al., 17 Jan 2025): Training uses AdamW, large-scale corpora, and standard learning rate warmup and decay schedules. In intra-word Transformer, typical settings are 4 layers, 12 attention heads, and model width $d=768$. Inter-word/sequence Transformer uses e.g., 12–36 layers and up to 7B parameters in large-scale settings (Neitemeier et al., 17 Jan 2025). Batch sizes and optimization schedules align with contemporary Transformer pretraining.

Hyperparameter details and scenario-specific parameters are tabulated below:

Component	Typical Settings	Source
Intra-word Transformer	4 layers, 12 heads, 768-dim	(Sun et al., 2023)
Inter-word Transformer	12 layers, 12 heads, 768-dim	(Sun et al., 2023)
Char Encoder/Decoder	23–55M param (1B–7B scale)	(Neitemeier et al., 17 Jan 2025)
Word Transformer	1.1B–9.2B params (1–7B scale)	(Neitemeier et al., 17 Jan 2025)
Loss	MLM or next-character cross-entropy	(Hwang et al., 2016, Sun et al., 2023)
Pretraining data	Wikipedia, BookCorpus, 1.2T bytes	(Sun et al., 2023, Neitemeier et al., 17 Jan 2025)

4. Empirical Results and Benchmarking

HCLMs have been thoroughly benchmarked for both language modeling and downstream task performance.

• Recurrent HCLM: On the One Billion Word Benchmark (Hwang et al., 2016), HLSTM-B with 4×1024 LSTM layers (34.9M parameters) achieves BPC=1.140 (PPL=60.7), outperforming traditional KN-5 (PPL=67.6) and large LSTM (PPL=93.3) with ≪2% of the parameter count.
• Transformer HCLM: On zero-shot evaluation (Table 1 below, excerpted from (Neitemeier et al., 17 Jan 2025)), hierarchical models at 1B, 3B, and 7B scale achieve equal or slightly superior performance to tokenizer-based baselines on most tasks, and superior performance (+68% rel.) on LAMBADA.

Scale	Model	DCLM word acc	MMLU	LAMBADA	HellaSwag	BoolQ
1B	Hierarchical	35.5%	26.0	56.5	46.5	60.2
1B	Baseline	35.3%	27.6	53.7	46.2	62.6

• General-Domain Robustness (Sun et al., 2023): HCLM achieves SQuAD 1.1 F1=90.4 and MNLI accuracy 84.4/84.3, outperforming CharacterBERT and CANINE under various types of input noise and domain shift (e.g., 10% char deletion, cross-domain NER). Accuracy degrades more gently versus BERT and other subword models.

Model	SQuAD 1.1 F1	MNLI-m/mm	MRPC Acc	W-NUT16 F1
BERT-Base	88.7	83.3/84.2	86.7	45.7
HCLM	90.4	84.4/84.3	88.2	47.9

• ASR Application (Hwang et al., 2016): With end-to-end character-level beam search, HLSTM-B (4×512, 8.48M params) reduces WER to 7.79% on WSJ eval92, outperforming larger LSTM baselines.
• Continued Pretraining and Cross-lingual Adaptation (Neitemeier et al., 17 Jan 2025): HCLMs process data nearly twice as fast during domain adaptation and retain more of original task performance under language/domain shift than tokenizer-based transformers.

5. Robustness, Generalization, and Open-Vocabulary Capacity

HCLMs exhibit marked robustness compared to subword/tokenizer-based models:

• Input Perturbation: HCLMs show 2–3× smaller accuracy drops under character substitution, deletion, permutation, or ALL-CAPS transforms in multiple zero-shot and QA/NLU tasks (Neitemeier et al., 17 Jan 2025, Sun et al., 2023).
• Domain Shift: HCLM NER F1 is consistently superior on out-of-domain datasets; rare or novel terms are handled seamlessly due to full character-level modeling.
• Ablations: Use of a dedicated [WORD_CLS] token for word-level summary achieves higher performance versus average/max pooling (Sun et al., 2023).
• Computational Efficiency: By compressing words into single embeddings before the deep sequence transformer, HCLMs run at inference speeds comparable to BERT-Base and up to twice that of other character-level alternatives such as CANINE (Sun et al., 2023).
• OOV and Open Vocabulary: Because all operations are character/byte-level at the interface, HCLMs generate and score any text without UNK tokens or fixed-size vocabularies (Hwang et al., 2016, Neitemeier et al., 17 Jan 2025).

6. Design Trade-offs and Extensions

HCLMs inherently balance local fine-grained modeling with global context:

• Compression vs. Expressiveness: Compute is focused where needed: shallow, efficient modules process intra-word signal, while deep, expressive context models handle higher-order dependencies. Very small intra-word encoders/decoders (~24M params) suffice even at large model scales (Neitemeier et al., 17 Jan 2025).
• Clocks and Resets: The use of hand-designed update schedules (clocks and resets) provides modularity and interpretability in recurrent HCLMs, with future work suggested for learnable gating mechanisms (Hwang et al., 2016).
• Segmentation Strategy: While whitespace splitting is competitive in alphabetic languages, Unicode-based segmenters offer greater universality in multilingual scenarios (Neitemeier et al., 17 Jan 2025).
• Generality: Hierarchical design enables direct extension to more than two levels (e.g., sentence, paragraph), although this is less explored.
• Task Universality: HCLMs have demonstrated applicability to language modeling, QA, NLU, ASR, and cross-lingual benchmarks, providing a unified framework across domains.

7. Comparative Perspectives and Future Directions

HCLMs bridge the divide between subword-limited models and surface‐level character models by providing (i) full open-vocabulary handling, (ii) hierarchical contextualization, and (iii) robustness to surface perturbations. Empirical findings indicate that HCLMs match or outperform prior architectures on many tasks, adapt more quickly to new domains/languages, and remain computationally efficient (Neitemeier et al., 17 Jan 2025, Sun et al., 2023, Hwang et al., 2016).

Potential future directions highlighted include learning the clock/reset schedules directly, growing hierarchies beyond word- and sentence-levels, and leveraging universal segmentation for non-alphabetic scripts. The hierarchical paradigm is positioned to serve as a general foundation for robust, flexible, and adaptable language processing systems.