Continuous Space Language Model (CSLM)

• CSLMs are probabilistic language models that map linguistic units into continuous latent spaces, enabling smooth modeling of semantic and syntactic dependencies.
• They employ architectures such as embedding-based LSTMs, continuous attention transformers, and autoregressive flow models to optimize language representation and generation.
• Empirical results demonstrate improved perplexity and efficient reasoning, while challenges remain in training stability and the extraction of symbolic abstractions.

A Continuous Space LLM (CSLM) is a probabilistic sequence model in which linguistic units—tokens, words, or subword entities—are represented and processed within a continuous, low-dimensional latent space. These models generalize traditional discrete LLMs by exploiting the representational and computational properties of continuous manifolds, enabling richer modeling of semantic and syntactic dependencies, facilitating optimization, and supporting novel paradigms for language generation, reasoning, and compression.

1. Mathematical Foundations and Model Classes

Continuous Space LLMs encompass several architectural classes unified by the parametrization of text as sequences or trajectories in a continuous vector space $\mathbb{R}^d$. The two principal settings are:

• Embedding-based CSLM: Discrete tokens are mapped to dense word embeddings via a learned map $\mathcal{E} : \mathcal{W} \rightarrow \mathbb{R}^d$; sequences then evolve according to (e.g.) recurrent or transformer-based neural dynamics, generating outputs through a softmax projection over the vocabulary (Chowdhury et al., 2020). The entire system acts on continuous representations throughout all layers except for the final token emission.
• Latent-variable CSLM: Text or speech is encoded into continuous latent variables (via VAEs, flows, or neural codecs), and these latents are then modeled autoregressively or via flows, often bypassing discrete tokenization completely, as in SLED for speech (Ma et al., 19 May 2025) or TarFlowLM (Zhang et al., 1 Jul 2025).

Recent theoretical work formalizes the CSLM as a Continuous State Machine (CSM), i.e., a smooth dynamical system $(M,U,T,s_0,O,\Delta)$ with latent state $s_t\in M\subset \mathbb{R}^d$ evolving under transition $T$ and emitting tokens via a probabilistic decoder $O$ (Wyss, 4 Dec 2025). Under regularity assumptions, the probabilistic evolution on $M$ admits a compact transfer operator $P$ with discrete spectral decomposition, giving rise to spectral partitions of semantic states.

2. Model Architectures and Training Paradigms

Embedding-LSTM CSLM

“Continuous-Space Neural LLMs” implemented via LSTMs (e.g., AWD-LSTM with weight dropping) operate by mapping tokens to embeddings $E \in \mathbb{R}^{|V| \times d}$ and processing these embeddings through multiple stacked recurrent layers. The output hidden state $h_t$ parametrizes next-token probabilities via

$$p(w_t|w_{1:t-1}) = \frac{\exp(h_t^\top V_{out}[:,w_t] + b_{w_t})}{\sum_{w'} \exp(h_t^\top V_{out}[:,w'] + b_{w'})}$$

(Chowdhury et al., 2020).

Robust generalization is achieved by DropConnect-style weight-dropping on the recurrent parameters and NT-ASGD optimization. Empirically, this yields substantial perplexity improvement on low-resource, morphologically rich languages by leveraging the smoothness of embedding space and strong regularization.

Transformer-Based Continuous Attention

Transformers can be extended to operate on trajectories in continuous time-space: $X: [0,T] \to \mathbb{R}^d$ with piecewise-constant or interpolated embedding functions. Attention mechanisms are formalized by integrals:

$$Y(t) = \int_0^t \frac{\exp(Q(t)^\top K(s)/\sqrt d)}{\int_0^t \exp(Q(t)^\top K(u)/\sqrt d)\,du} V(s)\,ds$$

This formalism allows the model to interpolate, shrink, or blend embeddings continuously, enabling expressiveness beyond discrete token boundaries (Marro et al., 4 Apr 2025).

Autoregressive Flows and Hierarchical Latents

TarFlowLM implements language modeling as density estimation in continuous latent space using stacks of invertible, transformer-parameterized autoregressive flows (Zhang et al., 1 Jul 2025). Token sequences are mapped to latent trajectories $\mathbf{z}_{1:T}$, which are processed via a hierarchy of normalizing flows (dimension-wise Mixture-CDF, token-wise Mixture-Rosenblatt), supporting alternation in temporal direction, block-wise token grouping, and multi-pass coarse-to-fine generation.

SLED models speech by encoding raw waveforms into continuous frame-wise representations (e.g., via Encodec), and then autoregressively predicts these latents using a transformer and a generator trained by minimizing energy distance, sidestepping quantization artifacts and hierarchical prediction (Ma et al., 19 May 2025).

In reasoning, CSLMs can maintain and evolve internal “continuous thoughts” as hidden representations, enabling feedback loops and parallel exploration of solution paths (Chain-of-Continuous-Thought, “Coconut” paradigm) (Hao et al., 2024).

3. Key Theoretical Properties

Modern analyses formalize CSLMs via spectral operator theory. Let $P:L^2(M,\mu)\to L^2(M,\mu)$ be the transfer operator on the latent manifold $M$. If $P$ is compact and admits a discrete spectrum, then:

• The leading eigenfunctions $\{\phi_i\}$ induce a finite partition (“spectral lumps”) of $M$ into basins of semantic invariance.
• Each spectral basin is definable in an o-minimal structure (logical tameness), coinciding with the spectral partition up to sets of measure zero (Wyss, 4 Dec 2025).

The Semantic Characterization Theorem guarantees that under ergodicity, boundedness, and smoothness assumptions on the model, discrete symbolic meanings emerge as invariant sets in the latent manifold.

Additionally, empirical results and visualization in (Marro et al., 4 Apr 2025) confirm that LLMs internally execute smooth continuous transitions across embedding space along both “time” (token duration) and “space” (embedding interpolation), which is not captured by discrete symbolic models.

4. Empirical Results and Applications

NLP and Speech Tasks

CSLMs have demonstrated superior empirical performance on a range of tasks:

• In Bengali language modeling, AWD-LSTM CSLMs achieved held-out perplexity of $51.2$, dramatically surpassing count-based $n$-gram and simple LSTM baselines (perplexity $860$, $227$, respectively), highlighting the benefit of dense embeddings and regularization in morphologically rich, low-resource contexts (Chowdhury et al., 2020).
• SLED achieves strong zero-shot and streaming speech synthesis performance on LibriSpeech-PC, matching or exceeding discrete token-based systems with a real-time factor $<$1 for inference, supporting both continuation and voice cloning schemes (Ma et al., 19 May 2025).

Reasoning and Search

The Coconut paradigm (Chain-of-Continuous-Thought) demonstrates that using feedback loops of continuous transformer hidden states enables more efficient planning, lower hallucination rates, and the emergence of breadth-first search over hypotheses in latent state space, outperforming chain-of-thought reasoning on complex proof tasks while using drastically fewer inference steps (Hao et al., 2024).

Flexible Generation and Modeling

TarFlowLM enables bi-directional context modeling through invertible flows, parallel block-wise generation, and hierarchical multi-pass decoding. On language modeling benchmarks such as Text8 and OpenWebText, it approaches the bits-per-character and perplexity of autoregressive transformers, demonstrating the capacity of flexible continuous-space generative models to match discrete token-based systems (Zhang et al., 1 Jul 2025).

5. Limitations, Open Problems, and Extensions

Current CSLMs face several challenges:

• Training latent-state feedback systems requires curriculum learning; naïve approaches often collapse or diverge (Hao et al., 2024).
• Excessive chaining of continuous latent steps may induce instability; careful control of latent feedback depth and proper use of discriminative value estimates are needed.
• Interpreting or extracting symbolic abstractions from continuous trajectories remains nontrivial, despite the formal guarantees of spectral and logical collapse to finite basins (Wyss, 4 Dec 2025).
• For speech and audio, scalability and representation richness of continuous latents must be balanced against computational cost in downstream models (Ma et al., 19 May 2025).

Extensions include hybrid training schemes interleaving discrete and continuous reasoning, reinforcement learning-based exploration of semantic basins, and the development of o-minimal, semantically controllable architectures for robust, interpretable generation and reasoning.

6. Connections to Classical and Implicit Models

An important distinction is that even purely discrete token-prediction models (standard transformers, LSTMs) are, in effect, implicitly continuous, as their entire computation—save for the final softmax—operates on smooth vector representations. Marro et al. (Marro et al., 4 Apr 2025) provide evidence that all major LLMs (Llama, Gemma, Phi, Mistral) construct continuous “time” and “space” internal mappings, admitting spectrum of outputs unattainable by strictly discrete models. Formal ties between continuous autoregressive flows and discrete AR models have been established by reduction under specific VAE/flow parametrizations (Zhang et al., 1 Jul 2025).

A plausible implication is that future LLMs will increasingly leverage the full flexibility and expressiveness of continuous latent computation, both for practical generative capabilities and for formal semantic interpretability.

7. Summary Table: Classes and Key Properties

Model Example	Representation	Training/Inference Mechanism
AWD-LSTM LM (Chowdhury et al., 2020)	Embeddings in $\mathbb{R}^d$	Recurrent, DropConnect, ASGD
Transformer, "Implicit" (Marro et al., 4 Apr 2025)	Piecewise $\mathbb{R}^d$-valued functions	Multi-layer attention, softmax output
TarFlowLM (Zhang et al., 1 Jul 2025)	Latents, flows in $\mathbb{R}^d$	Stacks of autoregressive normalizing flows, mixture decoding
SLED (speech) (Ma et al., 19 May 2025)	Audio frame latents	Transformer AR modeling, energy distance loss
Coconut (Hao et al., 2024)	Feedback in hidden state	Transformer, continuous latent reasoning loop
CSMs (theory) (Wyss, 4 Dec 2025)	Latent manifold $M$	Markov kernel, spectral operator theory

In conclusion, CSLMs span a spectrum from embedding-based neural models to advanced invertible-flow frameworks and theoretical dynamical systems. These structures enable robust modeling of linguistic phenomena while formalizing the emergence of discrete semantics from fundamentally continuous computation.