SONAR-LLM: Hybrid Semantic Transformer

• SONAR-LLM is a decoder-only autoregressive transformer that unifies semantic abstraction with likelihood-based training through a frozen SONAR decoder.
• It segments text into sentences and maps them to a 1024-dimensional embedding space, enabling efficient semantic sequence modeling and reduced trainable parameters.
• Empirical results demonstrate that SONAR-LLM achieves competitive performance in text generation and summarization tasks while enhancing long-context processing efficiency.

SONAR-LLM is a decoder-only autoregressive transformer that generates text by predicting a sequence of sentence-level embeddings in the continuous SONAR space and is supervised by token-level cross-entropy through a frozen SONAR decoder. This hybrid approach unifies the semantic abstraction of Large Concept Models (LCMs) with efficient, likelihood-based training. SONAR-LLM eliminates dependence on diffusion objectives, enables semantic sequence modeling, and achieves competitive performance across diverse text generation tasks. All model code and checkpoints are publicly available, supporting reproducibility and further research (Dragunov et al., 7 Aug 2025).

1. Model Architecture

SONAR-LLM operates on sentence-level semantic representations, utilizing a modular architecture with three primary components: sentence segmentation, frozen embedding/decoding modules (SONAR encoder/decoder), and the trainable transformer prior.

1.1 SONAR Embedding Space

Documents are segmented into sentences $s_1, s_2, \ldots$ using the NLTK Punkt tokenizer. Each sentence $s_t$ is mapped to a 1024-dimensional embedding $$\mathbf{e}_t = \mathcal{E}(s_t) \in \mathbb{R}^{1024}$$ via a frozen SONAR encoder $\mathcal{E}$ (from Paul-Ambroise Duquenne et al., 2023). Training and inference autoregressive prefixes are formed as $$\mathbf{e}_{<t} = (\mathbf{e}_1, \ldots, \mathbf{e}_{t-1})$$.

1.2 Decoder-only Transformer Prior

The transformer prior $f_\theta$ follows the same layer pattern as Llama 3, with rotary positional encodings and RMS-norm. Its input and output vocabulary each consist of a single continuous “concept” symbol (the sentence embedding). The autoregressive prediction is defined as $\hat{\mathbf{e}}_t = f_\theta(\mathbf{e}_{<t})$.

Model variants are produced across sizes from 39M to 900M parameters in the base LLM configuration. However, as both the SONAR encoder and decoder are frozen, the count of trainable parameters for SONAR-LLM is considerably lower for each configuration (e.g., 11M, 34M, 170M, 450M, 700M for increasing sizes).

1.3 Teacher Forcing and Positional Encoding

During training, teacher forcing is employed: ground-truth sentence embeddings are provided at each next step. Rotary encodings handle sentence-level positions.

2. Hybrid Training Objective

SONAR-LLM introduces a hybrid supervision mechanism that propagates token-level cross-entropy loss through the SONAR decoder while predicting in continuous embedding space.

2.1 Loss Computation

After outputting the next embedding $\hat{\mathbf{e}}_t$, it is fed through the frozen SONAR decoder $\mathcal{D}$, producing vocabulary logits $$\mathbf{z}_t = \mathcal{D}(\hat{\mathbf{e}}_t) \in \mathbb{R}^{|\mathcal{V}|}$$. Standard cross-entropy is computed against the ground-truth sentence $s_t$0 at the token level, with document-level loss:

$s_t$1

Gradients pass through $s_t$2 but leave SONAR frozen, directly aligning embedding predictions with tokenized content.

2.2 Comparison to LCM Objectives

Earlier LCM variants employed either mean-squared error (MSE) loss:

$s_t$3

or diffusion-based denoising objectives:

$s_t$4

The SONAR-LLM objective reinstates a single-step, likelihood-based training signal via token cross-entropy, circumventing the computational expense and inherent instability of diffusion sampling.

3. Scaling Behaviour and Ablation Studies

3.1 Empirical Scaling Laws

Validation loss $s_t$5 versus model size $s_t$6 (full LLM parameters) is fit with $s_t$7. Parameters for each approach are summarized below:

Model	$s_t$8	$s_t$9	$$\mathbf{e}_t = \mathcal{E}(s_t) \in \mathbb{R}^{1024}$$0
LLM	$$\mathbf{e}_t = \mathcal{E}(s_t) \in \mathbb{R}^{1024}$$1	$$\mathbf{e}_t = \mathcal{E}(s_t) \in \mathbb{R}^{1024}$$2	$$\mathbf{e}_t = \mathcal{E}(s_t) \in \mathbb{R}^{1024}$$3
SONAR-LLM	$$\mathbf{e}_t = \mathcal{E}(s_t) \in \mathbb{R}^{1024}$$4	$$\mathbf{e}_t = \mathcal{E}(s_t) \in \mathbb{R}^{1024}$$5	$$\mathbf{e}_t = \mathcal{E}(s_t) \in \mathbb{R}^{1024}$$6
MSE LCM	$$\mathbf{e}_t = \mathcal{E}(s_t) \in \mathbb{R}^{1024}$$7	$$\mathbf{e}_t = \mathcal{E}(s_t) \in \mathbb{R}^{1024}$$8	$$\mathbf{e}_t = \mathcal{E}(s_t) \in \mathbb{R}^{1024}$$9
Diff. LCM	$\mathcal{E}$0	$\mathcal{E}$1	$\mathcal{E}$2

All regressions exhibit $\mathcal{E}$3. The exponent for SONAR-LLM ($\mathcal{E}$4) is above other embedding-space methods, indicating enhanced scaling efficiency.

3.2 Ablations

Four ablation variants were evaluated with the same transformer prior:

1. Pure token-level LLM (token cross-entropy)
2. MSE LCM (embedding MSE)
3. Diffusion LCM (embedding diffusion loss)
4. SONAR-LLM (cross-entropy via frozen decoder)

No weight-mixing or multi-objective schedules were employed. Only the hybrid cross-entropy objective yielded performance surpassing embedding-only losses.

4. Empirical Performance and Benchmarking

4.1 Automatic Story Generation

In GPT-4o-based scoring on 512 TinyStories continuations, SONAR-LLM consistently surpassed both MSE and diffusion LCM on grammar, creativity, coherence, and plot consistency across all scales. The token-level LLM retained top performance overall.

4.2 Standard NLG Metrics

For 3rd-sentence prediction (under both short and long prompts), SONAR-LLM matched or slightly outperformed LLMs on BLEU, ROUGE-L, and METEOR metrics. Both MSE and diffusion LCM variants performed substantially worse.

4.3 Summarization (XSum & CNN/DM)

For summarization, models (1.3B parameters) pretrained on mixed corpora were evaluated on 512 samples:

Model	XSum ROUGE-L	XSum METEOR	CNN/DM ROUGE-L	CNN/DM METEOR
SONAR-LLM	19.3	15.2	16.0	10.4
LLM (beam)	18.7	15.4	18.3	16.5
LLM (greedy)	18.9	14.9	18.7	14.1
MSE LCM	12.2	8.7	7.6	3.7
Diff. LCM	12.0	8.3	10.2	5.1

SONAR-LLM outperformed sentence-level models and was competitive with token-level LLMs on XSum.

4.4 Qualitative Analysis

Continuations generated by SONAR-LLM (900M) demonstrated narrative coherence, semantic appropriateness, and fluent realization, evidencing effective semantic-level planning and token-level execution.

4.5 Inference Efficiency

Theoretically, for a 600M model and sentence length of 60 tokens:

• Token-level LLMs offer slightly lower FLOPs per token for short contexts.
• For input lengths beyond $\mathcal{E}$5 tokens, SONAR-LLM's per-sentence decoding achieves near-linear growth up to 1M tokens.
• Both architectures are quadratic in self-attention, but SONAR-LLM’s sentence-level compression markedly improves efficiency for extended contexts.

5. Implementation Methodology

5.1 Data and Preprocessing

Training utilized TinyStories, TinyTextbooks, Wikitext-103, XSum, CNN/DM, and web-crawled datasets. Sentences were segmented by NLTK Punkt. Each document appended a literal “End of sequence.” sentence, with stop criteria at inference defined by $\mathcal{E}$6 or maximum $\mathcal{E}$7 sentences.

5.2 Training Procedure

Training used up to 8 × A100 80GB GPUs, for 4 epochs per model. Learning rates were cosine-decayed; SONAR-LLM used an initial rate of $\mathcal{E}$8, while baselines used $\mathcal{E}$9. Teacher forcing on embeddings was applied, with a progressive schedule introducing 5% free-running sampling after epoch 3.

5.3 Reproducibility

All code, evaluation scripts, and checkpoints (covering TinyStories and summarization models) are released at https://github.com/FusionBrainLab/SONAR-LLM/tree/main.

6. Conceptual Implications and Significance

SONAR-LLM connects the representational efficiency of concept-level sequence models with the ground-truth fidelity of token-level supervision. By predicting sentence embeddings but backpropagating cross-entropy at the token level through a frozen decoder, the model preserves semantic abstraction, remains consistent with likelihood training, and improves scalability compared to previous LCM approaches. It matches or outperforms token-only LLMs in various text generation and summarization benchmarks, especially at longer context lengths where computational efficiency becomes critical (Dragunov et al., 7 Aug 2025).