MultiHashFormer: Hash-based Generative Language Models

Abstract: LLMs (LMs) represent tokens using embedding matrices that scale linearly with the vocabulary size. To constrain the parameter footprint, prior work proposes hashing many tokens into a single vector within encoder-only models. While this offers parameter efficiency, many-to-one collisions prevent its use in causal LMs. In this paper, we propose MultiHashFormer, a new framework that allows hash-based autoregression. Each token is represented as a unique hash signature, a short sequence of discrete hash IDs, generated by multiple independent hash functions. A Hash Encoder compresses this signature into a single latent vector for processing by a Transformer decoder. Then, a Hash Decoder generates the hash signature of the next token, which is then mapped back to text. We evaluate our approach at the 100M, 1B and 3B parameter scales, demonstrating that MultiHashFormer consistently outperforms standard Transformer LMs across multiple benchmarks. Furthermore, we show that our model handles multilingual vocabulary expansion with a constant parameter footprint without any modifications.

• The paper introduces a novel modular hash-based embedding that replaces conventional matrices with unique multi-ID hash signatures, ensuring collision-free generation.
• The paper demonstrates exponential vocabulary scalability via multiple hash functions while mitigating the softmax bottleneck and enhancing rare token expressiveness.
• The paper shows that multilingual vocabulary expansion is possible without additional parameters, achieving improved benchmark performance across various tasks.

MultiHashFormer: Hash-Based Generative LLMs

Motivation and Architectural Innovation

MultiHashFormer addresses the longstanding vocabulary bottleneck in generative LMs stemming from embedding matrices that scale linearly with vocabulary size. Prior hash-based approaches (e.g., HashFormer, Proformer) allow parameter-efficient token representation via many-to-one hash mappings, but suffer from token collisions that make autoregressive generation intractable—colliding tokens cannot be deterministically recovered from hash indices. Drawing on distributed memory theory and dynamic chaotic memory systems, MultiHashFormer replaces the conventional embedding matrix with a modular interface that maps each token to a unique multi-ID hash signature, which is a fixed-length tuple generated by multiple independent hash functions. This signature guarantees collision-free mapping for generative tasks and enables exponential scaling of vocabulary capacity with only sublinear parameter growth.

The framework consists of three modular components: a Hash Encoder that compresses multi-ID signatures into dense vectors, a Transformer-based Sequence Processing Backbone that produces contextualized representations, and a Hash Decoder that autoregressively reconstructs hash signatures for output tokens. Each hash function is implemented via MurmurHash3, and a rehash strategy ensures uniqueness for all tokens in the vocabulary. The compositional embedding is computed using learned gates, projecting multi-bucket embeddings to the backbone’s latent space.

Figure 1: Overview of the MultiHashFormer framework using three different hash functions and B buckets, constructing a unique multi-ID hash signature for each token.

Hash-Based Signature Compression and Sequence Modeling

MultiHashFormer’s key advance lies in multi-hash signature generation. For $H$ hash functions and $B$ buckets each, the theoretical vocabulary capacity is $B^H$, allowing support for vocabularies on the order of quadrillions without expanding model parameters. Each token’s signature is compressed via gated context-aware compositional embedding, and the Transformer backbone processes these representations identically to standard decoders.

The Hash Decoder employs a cascaded predictor—each hash coordinate is predicted sequentially via dedicated parameter slices. Soft embeddings are passed through a cascade mixer, recursively updating internal state and enabling structured error correction. At training, unconstrained decoding across all possible signature combinations optimizes a stricter subset corresponding to actual vocabulary tokens, with inference constrained to valid signature combinations to ensure semantic fidelity.

Empirical Evaluation and Numerical Results

MultiHashFormer was evaluated against standard Transformer-based LMs at 100M, 1B, and 3B parameter scales, using English-centric benchmarks (LAMBADA, ARC-E, COPA, OBQA, PIQA, HellaSwag, RACE, SciQ, SIQA, ReCoRD) and rare word similarity datasets (Card-660). Strong empirical results are reported:

• On 1B and 3B scales, MultiHashFormer consistently outperforms parameter-matched baselines on 9 of 11 tasks.
• On LAMBADA, MHF (H4B16K) achieves 37.26% accuracy at 3B scale, a +8.62% gain.
• On ReCoRD, MHF (H4B16K) reaches 64.90% at 1B, surpassing Standard+2L (63.78%).
• Card-660 rare word similarity correlations are highest with MultiHashFormer (second-to-last states: $r=0.32$, $\rho=0.32$ for H4B16K).

Performance analysis reveals that larger embedding space (higher $H$, $B$) boosts reading comprehension and reasoning. However, excessive bucket fragmentation may degrade rare factual token accuracy in long-tailed distributions. For small models (100M), parameter allocation favors transformer depth over embedding compression, but MultiHashFormer’s advantage scales with larger decoders.

Vocabulary Expansion: Practical Implications

MultiHashFormer uniquely enables multilingual vocabulary expansion without increasing parameter count, simply by registering new hash signatures. Experiments with expanded vocabularies (48K tokens, +15K from Arabic, Hindi, Chinese) show:

• MHF (H4B16K) at 1B and 3B scales outperforms Standard on 12/22 and 13/22 multilingual tasks, respectively.
• No catastrophic forgetting was observed relative to baselines.
• Standard models require 31M additional parameters for expansion, MultiHashFormer requires none.

(Figure 2)

Figure 2: MultiHashFormer’s performance after vocabulary expansion from 32K to 48K tokens, showing competitive or superior scores versus Standard models at 1B and 3B scales without parameter increases.

Theoretical Analysis: Mitigating the Softmax Bottleneck

MultiHashFormer elevates the rank of the output log-probability matrix by decomposing token-level prediction into a product of hash coordinate probabilities, effectively mitigating the softmax bottleneck inherent to classical LMs. The rank upper bound for MultiHashFormer is $\min(B, H \times d)$, compared to $d$ for Standard baselines. As a result, it improves model expressiveness for high-entropy tasks, notably LAMBADA and HellaSwag.

Comprehensive ablation demonstrates that multi-ID signatures are essential—Single-ID hash mappings with increased bucket count cannot match the performance of multi-hash configurations. The model’s efficacy is further validated by stability across different hash functions and the negligible impact of locality-sensitive hashing.

Implications and Future Directions

The practical implication is that MultiHashFormer provides an efficient pathway for scaling LMs in multilingual and domain-adaptive settings, decoupling vocabulary growth from parameter expansion. Theoretically, the framework introduces a robust mitigation for the softmax bottleneck in large vocabulary regimes and offers superior semantic compression for rare tokens. With architectural modularity, any backbone can be plugged in, and the hash interface remains language-agnostic.

Future developments should focus on scaling MultiHashFormer to larger (>7B) model sizes, integrating more advanced compositional gates, and exploring adaptive hash allocation for dynamic vocabulary environments. Direct integration with token-free and byte-level modeling could further enhance parameter efficiency. MultiHashFormer’s modular approach may inspire new directions in distributed representation learning and scalable generative modeling.

Conclusion

MultiHashFormer achieves parameter-efficient generative modeling via collision-free multi-hash signatures, outperforming standard Transformer LMs on diverse tasks and enabling seamless multilingual vocabulary expansion without additional parameters. Its mitigation of the vocabulary bottleneck and softmax bottleneck provides both practical utility for continual learning and theoretical advantages in model expressivity.