---
title: 'MultiHashFormer: Hash-Based Language Model'
url: https://www.emergentmind.com/topics/multihashformer
type: topic
---

# MultiHashFormer: Hash-Based Language Model

Searching arXiv for MultiHashFormer and closely related hash-based language modeling work.
MultiHashFormer is a decoder-only language modeling framework introduced in “MultiHashFormer: Hash-based Generative Language Models” [2606.28057]. It replaces the standard token-indexed embedding layer and output softmax head with a hash-based interface designed for autoregressive generation. Instead of assigning each token a single discrete ID, it assigns each token a unique multi-hash signature, a short sequence of hash IDs produced by multiple independent hash functions. A Hash Encoder compresses that signature into a latent vector for a Transformer decoder, and a Hash Decoder predicts the next token’s signature coordinate by coordinate before mapping the predicted signature back to text. The method is motivated by the claim that conventional embedding and output matrices scale linearly with vocabulary size, creating a vocabulary bottleneck, while earlier hash-based methods were unsuitable for causal language modeling because many-to-one collisions made deterministic decoding impossible [2606.28057].

## 1. Vocabulary bottleneck and the generative hashing problem

In a standard language model, the input embedding matrix is typically
$$
\mathbf E \in \mathbb R^{|\mathcal V|\times d},
$$
and the output projection is
$$
\mathbf W_o \in \mathbb R^{d\times |\mathcal V|}.
$$
Even with weight tying, the token interface remains indexed by vocabulary items, so its parameter footprint scales linearly with $|\mathcal V|$. MultiHashFormer presents this as a vocabulary bottleneck: adding tokens, adapting to new domains, or extending to new languages usually requires enlarging token-specific parameter matrices [2606.28057].

Earlier compression schemes proposed hashing many tokens into a smaller set of shared buckets. That reduces the number of physical embedding vectors, but prior hash-based token representations generally use many-to-one mappings, so multiple tokens can share the same hash bucket. In encoder-only or discriminative settings this can remain usable, because the model only needs a representation for classification or retrieval. In causal language modeling, however, if the decoder predicts a hashed index shared by several tokens, there is no deterministic way to know which token should be emitted. MultiHashFormer is proposed specifically to remove that collision ambiguity in autoregressive generation [2606.28057].

Its central design move is to replace the single colliding hash ID with a multi-hash signature. For a token $w$, the signature is
$$
[\mathcal H_1(w), \mathcal H_2(w), \dots, \mathcal H_H(w)],
$$
where each $\mathcal H_i$ is an independent hash function and each coordinate lies in $\{0,\dots,B-1\}$, with $0$ reserved for padding. Individual coordinates may collide across tokens, but the full signature is constructed to be collision-free within the actual vocabulary. This converts generative decoding from ambiguous single-bucket prediction into deterministic signature prediction [2606.28057].

The paper emphasizes that the virtual signature space scales combinatorially as $B^H$. As an illustration, it states that $H=4$ and $B=16{,}000$ yield a theoretical upper bound around $16000^4 \approx 6.5\times 10^{16}$ distinct signatures, while parameter cost depends on $H$, $B$, and $d$, not on the actual vocabulary size once the tables are fixed. This suggests that the method is intended not only as a compression device but also as a way to decouple vocabulary capacity from token-indexed parameter growth [2606.28057].

## 2. Architectural organization

MultiHashFormer consists of three modules: a Hash Encoder, a sequence-processing backbone implemented as a standard decoder-only Transformer, and a Hash Decoder [2606.28057].

The token-to-signature mapping uses $H$ independent hash functions. For non-padding tokens, the $i$-th coordinate is
$$
\mathcal H_i(w) = \mathrm{MMH3}(w, \mathrm{seed}_i)\pmod{B-1} + 1 \quad \text{if } w \neq \mathrm{pad},
$$
and padding is assigned
$$
\mathcal H_i(\mathrm{pad}) = 0.
$$
The construction uses MurmurHash3 and an iterative rehashing procedure: if a newly added token collides with an existing token’s full signature, the seed of the final hash function $\mathrm{seed}_H$ is modified until an unused signature is found. The resulting full-signature mapping is therefore collision-free over the actual vocabulary [2606.28057].

The Hash Encoder uses $H$ separate embedding matrices,
$$
\mathbf E^{(i)} \in \mathbb R^{B\times d}, \qquad i\in\{1,\dots,H\}.
$$
For token $w$, the encoder retrieves one row from each table and combines them through a gated compositional embedding:
$$
\mathbf{e} = \left( \sum_{i=1}^H \tilde{\alpha}_i \mathbf{E}^{(i)}_{[\mathcal{H}_i(w), :]} \right) \mathbf{W}_{s},
$$
with normalized weights
$$
\tilde{\alpha}_i = \frac{\exp^{\alpha_i}}{\sum_{j=1}^H \exp^{\alpha_j}},
$$
and
$$
\alpha_i = \sigma \left( \mathbf{E}^{(i)}_{[\mathcal{H}_i(w), :]} \mathbf{W}_{1} \right) \mathbf{W}_{2}.
$$
The paper notes typesetting issues, but it states that the intended structure is a bottleneck MLP producing a scalar gate for each coordinate, followed by a softmax over coordinates and a learned linear adaptation. The bottleneck dimension is $d_z \ll d$, and in experiments $d_z=64$ [2606.28057].

The backbone is otherwise conventional. For an input sequence $(w_1,\dots,w_n)$, the encoder outputs $\mathbf e_i \in \mathbb R^d$, which are stacked as
$$
\mathbf X = [\mathbf e_1,\mathbf e_2,\dots,\mathbf e_n]^\top \in \mathbb R^{n\times d}.
$$
A standard causal Transformer with $L$ layers processes this sequence, and experiments use a Qwen3-style decoder with RoPE; the appendix reports $\text{rope theta}=1\text{M}$. The reported context length is $2048$ [2606.28057].

The Hash Decoder predicts the next token’s signature sequentially across the $H$ coordinates. At time $t$, it initializes with the contextualized state
$$
\mathbf c^{(1)} = \mathbf h_t.
$$
For each coordinate $i$, it predicts bucket logits
$$
\mathbf o^{(i)} = \mathbf W_o^{(i)\top}\mathbf c^{(i)},
$$
where
$$
\mathbf W_o^{(i)} \in \mathbb R^{d\times B}.
$$
The model ties input and output weights:
$$
\mathbf W_o^{(i)\top} = \mathbf E^{(i)}.
$$
For non-terminal heads, it computes
$$
\mathbf p^{(i)} = \mathrm{Softmax}(\mathbf o^{(i)}),
$$
then forms a soft expected embedding
$$
\mathbf e^{(i)} = \mathbf E^{(i)\top}\mathbf p^{(i)},
$$
and updates the cascade state with a low-rank bottleneck residual mixer:
$$
\mathbf{c}^{(i+1)} = \mathbf{c}^{(i)} +\mathbf{W}_{up}^{(i)\top}\sigma(\mathbf{W}_{dn}^{(i)\top}\begin{bmatrix} \mathbf{c}^{(i)} \\ \mathbf{e}^{(i)}\end{bmatrix}).
$$
This means later coordinate predictions are conditioned on earlier ones rather than being predicted independently [2606.28057].

## 3. Probabilistic formulation, decoding, and expressiveness

During training, MultiHashFormer factorizes token probability into the product of coordinate probabilities:
$$
P_{train}(w \mid \cdot) = \prod_{i=1}^{H} P(\mathcal{H}_i(w) \mid \cdot).
$$
The paper does not present the loss as an explicit summed cross-entropy expression, but it states that the implied objective is the negative log-likelihood of the target coordinates,
$$
-\log P_{train}(w\mid\cdot) = -\sum_{i=1}^{H}\log P(\mathcal H_i(w)\mid\cdot).
$$
No label smoothing, auxiliary losses, or special regularization terms are described [2606.28057].

At inference, invalid signatures must be excluded. MultiHashFormer therefore renormalizes over the actual vocabulary only:
$$
P_{inf}(w \mid \cdot) = \frac{\prod_{i=1}^{H} P(\mathcal H_i(w)\mid \cdot)}{\sum_{w' \in \mathcal{V}_{actl}} \prod_{i=1}^{H} P(\mathcal H_i(w') \mid \cdot)}.
$$
Operationally, the model computes the log-score of each valid token by summing the log-probabilities of the hash IDs in its registered signature and then applies a softmax over valid tokens only. The paper does not describe approximate nearest-neighbor search or error-correction heuristics; decoding is exact under the assumption that every actual vocabulary item has a unique registered signature and inference is restricted to those valid signatures [2606.28057].

This training–inference distinction is structurally important. Training assigns probability mass over the entire virtual signature space, including signatures that do not correspond to registered tokens, whereas inference explicitly prunes invalid signatures. A plausible implication is that the model’s output layer is no longer a direct vocabulary classifier in the conventional sense; instead, it is a structured predictor over hash coordinates whose outputs must be reassembled over valid lexical items.

The paper also places MultiHashFormer in relation to the softmax bottleneck. For a standard language model with hidden-state matrix $\mathbf H_\theta$ and output matrix $\mathbf W_\theta$, the rank of $\mathbf H_\theta \mathbf W_\theta^\top$ is upper-bounded by $d$. The appendix argues that MultiHashFormer raises the rank upper bound of the modeled conditional distribution roughly to
$$
\mathrm{rank}(\hat{\mathbf A}_{\text{MHF}})\le \min(B, H\times d),
$$
up to notation issues in the appendix. The claim is not that the model can represent arbitrary rank, but that the multi-head hash factorization can increase expressiveness relative to a single standard softmax [2606.28057].

## 4. Parameter scaling, optimization, and vocabulary expansion

The main parameter-scaling claim is that standard tied embeddings require token-parameter growth on the order of
$$
O(|\mathcal V|d),
$$
whereas MultiHashFormer’s input/output hash tables scale on the order of
$$
O(HBd),
$$
plus small gating and cascade-mixer parameters. This is the basis for the paper’s statement that vocabulary expansion can occur with a constant parameter footprint, provided the chosen $(H,B)$ configuration still has sufficient unused signature capacity [2606.28057].

The reported pretraining setup uses a subset of English FineWeb-Edu. The 100M models are trained on 10B tokens, while the 1B and 3B models are trained on 100B tokens. The context length is 2048, the global batch size is 256, the optimizer is AdamW, precision is BF16, the scheduler is cosine, warmup is 2000 steps, weight decay is 0.1, and gradient clipping is 1.0. The tokenizer is the Mistral-7B-v0.3 BPE tokenizer with vocabulary size 32,768 [2606.28057].

The backbone sizes are reported as follows: the 100M model uses hidden size 768, 12 layers, and 12 heads; the 1B model uses hidden size 2048, 20 layers, and 16 heads; the 3B model uses hidden size 2048, 36 layers, and 16 heads. The principal MultiHashFormer configurations are H3B10K, with $H=3$ and $B=10{,}240$ in the main text, and H4B16K, with $H=4$ and $B=16{,}384$. The appendix contains a minor inconsistency, stating 10,624 for the H3B10K abbreviation table [2606.28057].

The vocabulary-expansion experiment is one of the paper’s main practical demonstrations. The authors continue pretraining on 6B multilingual tokens from Arabic, Chinese, and Hindi, plus 2B English, while adding 5K tokens per language for a final vocabulary of 48,122. In this setting, MultiHashFormer adds zero parameters, whereas the standard baselines must add 31 million additional parameters for the new tokens. The paper states that no architectural modifications are required for MultiHashFormer under this expansion [2606.28057].

This constant-footprint property is qualified in the paper itself. It holds only if the existing virtual signature space remains sufficiently large and the fixed bucket tables remain adequate for the expanded vocabulary. This suggests that the method is best understood as shifting the scaling frontier rather than eliminating it absolutely.

## 5. Empirical behavior and ablation evidence

The evaluation covers language modeling, commonsense reasoning, reading comprehension, rare-word representation, and multilingual vocabulary expansion. Benchmarks include LAMBADA; ARC-E, COPA, OBQA, PIQA, and HellaSwag; RACE, SciQ, and SIQA; and ReCoRD [2606.28057].

At the 100M scale, MultiHashFormer does not consistently outperform the baselines. The paper explicitly states that the Standard+4L model is often stronger. On LAMBADA, the reported scores are Standard 15.33, Standard+4L 19.64, MHF H3B10K 16.40, and MHF H4B16K 18.79. The authors interpret this as evidence that at small scale decoder depth matters more than the vocabulary interface [2606.28057].

At 1B scale, the reported results are substantially stronger. Selected values from Table 1 are: LAMBADA 30.41 for Standard, 30.66 for Standard+2L, 35.78 for MHF H3B10K, and 35.34 for MHF H4B16K; ARC-E 59.85, 60.23, 61.95, and 64.35; COPA 64.00, 65.00, 65.00, and 71.00; PIQA 65.78, 65.02, 68.17, and 66.49; HellaSwag 38.99, 39.85, 40.45, and 41.32; RACE 29.38, 29.28, 29.00, and 31.39; SciQ 69.90, 72.10, 68.30, and 70.20; SIQA 39.10, 38.89, 40.48, and 41.66; and ReCoRD 63.20, 63.78, 64.51, and 64.90. The paper highlights that, under strict parameter matching, both MultiHashFormer variants outperform Standard and often outperform Standard+2L [2606.28057].

At 3B scale, the H4B16K configuration outperforms Standard on 9 of 11 tasks. The paper reports, for example, LAMBADA improving from 28.64 to 37.26, ARC-E from 60.23 to 66.88, COPA from 65.00 to 68.00, OBQA from 31.80 to 34.60, PIQA from 65.83 to 68.39, HellaSwag from 39.65 to 42.89, RACE from 29.76 to 30.91, SciQ from 64.80 to 70.60, SIQA from 38.28 to 40.74, and ReCoRD from 62.26 to 64.13. The paper treats the large LAMBADA gain as a particularly strong indicator of improved context-dependent next-word discrimination [2606.28057].

The rare-word analysis on Card-660 reports improved correlation with human judgments at 1B scale. Using second-to-last hidden states, the paper gives Pearson/Spearman values of 0.26/0.25 for Standard, 0.32/0.29 for MHF H3B10K, 0.29/0.29 for Standard+2L, and 0.30/0.32 for MHF H4B16K. The appendix examples are described as showing higher similarity for abbreviations, aliases, misspellings, and synonyms [2606.28057].

The ablation studies directly target the paper’s core claim that multi-ID hashing is essential for generative use. On LAMBADA, the comparison between single-hash and multi-hash variants is: H1B4K 4.29 versus H4B4K 30.27, H1B8K 8.77 versus H4B8K 30.47, and H1B16K 14.30 versus H4B16K 30.91. The paper presents this as strong evidence that simply increasing the bucket count of a single-ID hash does not solve the generative collision problem. Additional ablations show that H4B16K is the best balance of efficiency and accuracy among the tested configurations, and that replacing MurmurHash3 with locality-sensitive hashing does not yield consistent improvement on LAMBADA [2606.28057].

## 6. Position within the literature, scope, and limitations

MultiHashFormer is specifically a hash-based generative language model. It should not be conflated with “MHFormer: Multi-Hypothesis Transformer for 3D Human Pose Estimation,” which is a video-based 3D human pose estimation method for monocular 2D-to-3D lifting and addresses an unrelated inverse problem in human pose estimation [2111.12707]. It also should not be conflated with “SemHash-LLM: A Multi-Granularity Semantic Hashing Framework for Document Deduplication,” whose proposed method is explicitly called SemHash-LLM and is framed around document deduplication rather than autoregressive language modeling; the queried term “MultiHashFormer” does not literally appear in that paper [2607.01601].

Within its own domain, MultiHashFormer’s principal strengths are stated as follows: it makes hash-based generation possible by using unique multi-ID signatures; it supports vocabulary expansion without changing hash table sizes when capacity suffices; it improves performance at 1B and 3B scales on multiple reasoning and reading benchmarks; it improves rare-word representations; and it is tokenizer-agnostic and language-agnostic in the sense that it does not rely on character- or morphology-specific heuristics [2606.28057].

The limitations enumerated in the paper are equally clear. Evaluation is reported only up to 3B parameters, not at 7B or above. Results are based on single-seed runs due to compute constraints. The paper does not provide detailed runtime or latency analysis, even though inference requires signature-to-token renormalization over the valid vocabulary. Performance at 100M is not consistently better than strong standard baselines. There are notation and formatting inconsistencies, especially around exact tensor shapes and one hash-configuration value. The method still assumes a discrete tokenizer and a registered vocabulary; it is not token-free [2606.28057].

Taken together, these properties position MultiHashFormer as a reparameterization of the token interface in decoder-only language models. Its novelty is not merely that it compresses embeddings, but that it replaces one-token/one-vector lookup and one-token/one-logit prediction with a collision-free multi-hash signature system that remains compatible with exact autoregressive decoding. This suggests that its main significance lies at the intersection of parameter-efficient lexical interfaces, vocabulary scaling, and structured alternatives to the conventional softmax head [2606.28057].

Source: https://www.emergentmind.com/topics/multihashformer