Word2Spike: Spike-Based Word Embeddings

• Word2Spike is a method that converts continuous embeddings into spike-based attractor states, acting as a bridge between modern embedding models and neuromorphic architectures.
• It employs techniques like absmean quantization combined with Poisson rate coding to preserve semantic structure while enabling sparse, event-driven processing.
• Empirical results demonstrate near-lossless reconstruction accuracy and significant energy efficiency gains, underscoring its viability for neuromorphic NLP applications.

Word2Spike denotes the conversion of linguistic representations into spike-based signals for spiking neural networks (SNNs), and more specifically names a 2025 method that maps continuous word embeddings into spike-based attractor states by combining absmean quantization with Poisson rate coding (Kalra et al., 9 Sep 2025). In the broader neuromorphic NLP literature, the same problem appears under several formulations: TF-IDF-derived document spike trains learned with STDP, deterministic or stochastic rate coding of pretrained word embeddings, Poisson conversion of normalized embeddings for converted SNN text classifiers, and spike-native Transformer or decoder pipelines that require token- or character-level inputs to be represented as binary or near-binary events (Białas et al., 2020, Knipper et al., 2024, Lv et al., 2024, Xing et al., 2024, Beger et al., 10 Jun 2026). Across these formulations, the unifying objective is to preserve useful semantic structure while replacing dense floating-point computation with sparse, event-driven processing.

1. Terminological scope and research lineage

In its narrow sense, Word2Spike refers to the mechanism introduced in “Word2Spike: Poisson Rate Coding for Associative Memories and Neuromorphic Algorithms,” where a continuous embedding is quantized into a ternary code and then converted, dimension by dimension, into Poisson spike trains intended to serve as spike-based attractor states in associative memory systems (Kalra et al., 9 Sep 2025). The paper positions the representation as a bridge between modern embedding models and neuromorphic architectures such as Hopfield-like networks, spiking transformers, and liquid state machines.

In a broader literature sense, “Word2Spike” may be treated as an Editor’s term for a family of word- or token-level text-to-spike pipelines. Earlier work on biologically plausible text representation transformed TF-IDF-weighted documents into parallel word-indexed spike trains and learned low-dimensional encodings with a two-layer SNN trained by a modified STDP rule (Białas et al., 2020). Later work on SNN NLP compared binary embeddings, Poisson rate coding, and deterministic rate coding for word- and sentence-level inputs, explicitly identifying text encoding as the key obstacle in bringing NLP to the neuromorphic setting (Knipper et al., 2024). A conversion-plus-fine-tuning line of work then showed how pretrained word embeddings can be normalized into $[0,1]$, converted into Poisson spike trains, and used in SNN versions of TextCNN for English and Chinese text classification (Lv et al., 2024).

This lineage suggests that Word2Spike is not a single architectural doctrine but a representational interface problem. The interface has been studied in unsupervised document encoding, supervised classification, associative memory, and increasingly in spike-driven Transformer-style sequence models.

2. Encoding mechanisms and mathematical formulations

The named Word2Spike method begins with a continuous embedding

$\vec{W} = [w_1, w_2, \ldots, w_n].$

It computes the mean absolute value

$\gamma = \frac{1}{n}\sum_{i=1}^{n} |w_i|$

and applies absmean quantization inspired by BitNet b1.58: $$\tilde{W}_i = \begin{cases} +1 & \text{if } w_i > \gamma, \ 0 & \text{if } |w_i| \leq \gamma, \ -1 & \text{if } w_i < -\gamma. \end{cases}$$ Each quantized dimension is then assigned one spiking neuron, with firing rates

$$\nu_i = \begin{cases} 100\,\text{Hz} & \text{if } \tilde{W}_i = +1, \ 0\,\text{Hz} & \text{if } \tilde{W}_i = 0, \ 50\,\text{Hz} & \text{if } \tilde{W}_i = -1. \end{cases}$$

Spikes are generated by a Poisson process over a fixed observation window, exemplified by $200$ ms, and decoding recovers the ternary representation via a threshold rule with a $72$ Hz boundary between the $-1$ and $+1$ rate levels (Kalra et al., 9 Sep 2025). The explicit design objective is one-to-one mapping at the quantized level under stochastic spiking noise.

Other Word2Spike-style systems use materially different encoders. In SNNLP, the proposed deterministic self-accumulate-and-fire (SAF) mechanism replaces stochastic Poisson sampling with repeated accumulation of the same floating-point value: $V_i(t) = V_i(t-1) + x_i(t)$ and

$\vec{W} = [w_1, w_2, \ldots, w_n].$0

The paper contrasts this with stochastic Poisson conversion and presents the SAF neuron as a more faithful rate-based encoding of the underlying floating-point magnitude (Knipper et al., 2024).

In the earlier biologically plausible document encoder, the input is not a pretrained embedding but a TF-IDF document-term vector. For each word relevance weight $\vec{W} = [w_1, w_2, \ldots, w_n].$1, a spike is generated at every millisecond during document presentation with probability

$\vec{W} = [w_1, w_2, \ldots, w_n].$2

with $\vec{W} = [w_1, w_2, \ldots, w_n].$3 ms, $\vec{W} = [w_1, w_2, \ldots, w_n].$4 ms, and $\vec{W} = [w_1, w_2, \ldots, w_n].$5 in the reported experiments (Białas et al., 2020). Here the spike train directly reflects weighted lexical salience rather than a dense semantic embedding.

In the converted TextCNN approach, pretrained embeddings are first clipped into $\vec{W} = [w_1, w_2, \ldots, w_n].$6, normalized by

$\vec{W} = [w_1, w_2, \ldots, w_n].$7

and shifted into $\vec{W} = [w_1, w_2, \ldots, w_n].$8; each scalar component is then converted into a Poisson spike train with firing rate proportional to its value (Lv et al., 2024). The literature therefore shows that Word2Spike is not synonymous with a single coding rule. Stochastic rate coding, deterministic rate coding, and TF-IDF-driven stochastic encoding all occur under the same general representational agenda.

3. Neural architectures that consume Word2Spike representations

The architectural consumer strongly shapes what “Word2Spike” must preserve. In the biologically plausible text representation setting, the input layer contains one neuron per dictionary word, the encoder layer contains $\vec{W} = [w_1, w_2, \ldots, w_n].$9 learned representation neurons, and an additional inhibitory neuron enforces competitive learning through winner-takes-all dynamics (Białas et al., 2020). The neuron model is leaky integrate-and-fire (LIF), synapses are updated by a modified STDP rule, and the final representation is formed from spike counts or spike rates per output neuron before classification by logistic regression.

SNNLP adopts a much simpler supervised architecture: ANN and SNN baselines share the same fully connected feedforward structure, with input size $\gamma = \frac{1}{n}\sum_{i=1}^{n} |w_i|$0 for floating-point word embeddings, $\gamma = \frac{1}{n}\sum_{i=1}^{n} |w_i|$1 for binarized word embeddings, or $\gamma = \frac{1}{n}\sum_{i=1}^{n} |w_i|$2 for sentence embeddings, followed by hidden layers of $\gamma = \frac{1}{n}\sum_{i=1}^{n} |w_i|$3 and $\gamma = \frac{1}{n}\sum_{i=1}^{n} |w_i|$4 neurons (Knipper et al., 2024). The SNN is trained with SLAYER, the inference window is typically $\gamma = \frac{1}{n}\sum_{i=1}^{n} |w_i|$5 ms per sample, and classification is inferred from the output neuron that spikes most often over the inference window.

The converted TextCNN line addresses variable-length text by retaining a convolution-plus-pooling backbone while replacing max-pooling with average-pooling, removing biases, converting embeddings into positive values, and then replacing ANN units with LIF neurons (Lv et al., 2024). After conversion, the model is fine-tuned by surrogate-gradient BPTT. This design separates sequence-length handling, which remains in the CNN backbone, from spike simulation time, which is handled by the SNN over $\gamma = \frac{1}{n}\sum_{i=1}^{n} |w_i|$6 timesteps.

More recent work embeds Word2Spike into Transformer-style computation. SpikeLM keeps standard learned token embeddings but converts hidden representations into elastic bi-spikes, allowing $\gamma = \frac{1}{n}\sum_{i=1}^{n} |w_i|$7 or $\gamma = \frac{1}{n}\sum_{i=1}^{n} |w_i|$8 spike values while preserving addition-based computation (Xing et al., 2024). SpikeDecoder formulates and compares embedding methods for a GPT-style decoder-only model, then moves an LIF neuron to the start of the spiking multi-head attention block so that even floating-point embeddings can be mapped into spikes at the decoder entrance (Beger et al., 10 Jun 2026). In all of these systems, the encoding is not an isolated pre-processing step; it is constrained by downstream neuron dynamics, residual design, normalization, and the requirement that later computation remain event-driven.

4. Empirical performance and reported advantages

The named Word2Spike paper evaluates original continuous embeddings, quantized embeddings, and spike-based embeddings on four criteria: SimLex-999, analogy accuracy, nearest-neighbor consistency, and reconstruction accuracy (Kalra et al., 9 Sep 2025). The reported main results are summarized below.

Representation	Benchmark	Reported result
Original	SimLex-999 (Spearman’s $\gamma = \frac{1}{n}\sum_{i=1}^{n} |w_i|$9)	0.540
Quantized	SimLex-999 (Spearman’s $$\tilde{W}_i = \begin{cases} +1 & \text{if } w_i > \gamma, \ 0 & \text{if } |w_i| \leq \gamma, \ -1 & \text{if } w_i < -\gamma. \end{cases}$$0)	0.542
Spike-based	SimLex-999 (Spearman’s $$\tilde{W}_i = \begin{cases} +1 & \text{if } w_i > \gamma, \ 0 & \text{if } |w_i| \leq \gamma, \ -1 & \text{if } w_i < -\gamma. \end{cases}$$1)	0.526
Original / Quantized / Spike-based	Analogy Accuracy (25 analogies)	37.50%
Original / Quantized	Overlap@10	0.885
Spike-based	Overlap@10	0.727
Spike-based	Reconstruction Accuracy	100.00%

The same paper also reports a preliminary setting with a $$\tilde{W}_i = \begin{cases} +1 & \text{if } w_i > \gamma, \ 0 & \text{if } |w_i| \leq \gamma, \ -1 & \text{if } w_i < -\gamma. \end{cases}$$2 ms window and rates $$\tilde{W}_i = \begin{cases} +1 & \text{if } w_i > \gamma, \ 0 & \text{if } |w_i| \leq \gamma, \ -1 & \text{if } w_i < -\gamma. \end{cases}$$3 Hz and $$\tilde{W}_i = \begin{cases} +1 & \text{if } w_i > \gamma, \ 0 & \text{if } |w_i| \leq \gamma, \ -1 & \text{if } w_i < -\gamma. \end{cases}$$4 Hz, yielding $$\tilde{W}_i = \begin{cases} +1 & \text{if } w_i > \gamma, \ 0 & \text{if } |w_i| \leq \gamma, \ -1 & \text{if } w_i < -\gamma. \end{cases}$$5 reconstruction accuracy, $$\tilde{W}_i = \begin{cases} +1 & \text{if } w_i > \gamma, \ 0 & \text{if } |w_i| \leq \gamma, \ -1 & \text{if } w_i < -\gamma. \end{cases}$$6 semantic similarity preservation, and $$\tilde{W}_i = \begin{cases} +1 & \text{if } w_i > \gamma, \ 0 & \text{if } |w_i| \leq \gamma, \ -1 & \text{if } w_i < -\gamma. \end{cases}$$7 nearest-neighbor consistency (Kalra et al., 9 Sep 2025). These figures support the paper’s core claim that the embedding-to-spike conversion is near-lossless at the quantized level.

In SNNLP, deterministic rate coding materially outperforms Poisson rate coding at the word level. For IMDb sentiment classification, word-level inference accuracy rises from $$\tilde{W}_i = \begin{cases} +1 & \text{if } w_i > \gamma, \ 0 & \text{if } |w_i| \leq \gamma, \ -1 & \text{if } w_i < -\gamma. \end{cases}$$8 with SNN-rate-rand to $$\tilde{W}_i = \begin{cases} +1 & \text{if } w_i > \gamma, \ 0 & \text{if } |w_i| \leq \gamma, \ -1 & \text{if } w_i < -\gamma. \end{cases}$$9 with SNN-rate; for CARER emotion classification, accuracy rises from $$\nu_i = \begin{cases} 100\,\text{Hz} & \text{if } \tilde{W}_i = +1, \ 0\,\text{Hz} & \text{if } \tilde{W}_i = 0, \ 50\,\text{Hz} & \text{if } \tilde{W}_i = -1. \end{cases}$$0 to $$\nu_i = \begin{cases} 100\,\text{Hz} & \text{if } \tilde{W}_i = +1, \ 0\,\text{Hz} & \text{if } \tilde{W}_i = 0, \ 50\,\text{Hz} & \text{if } \tilde{W}_i = -1. \end{cases}$$1, and MRR from $$\nu_i = \begin{cases} 100\,\text{Hz} & \text{if } \tilde{W}_i = +1, \ 0\,\text{Hz} & \text{if } \tilde{W}_i = 0, \ 50\,\text{Hz} & \text{if } \tilde{W}_i = -1. \end{cases}$$2 to $$\nu_i = \begin{cases} 100\,\text{Hz} & \text{if } \tilde{W}_i = +1, \ 0\,\text{Hz} & \text{if } \tilde{W}_i = 0, \ 50\,\text{Hz} & \text{if } \tilde{W}_i = -1. \end{cases}$$3 (Knipper et al., 2024). The paper summarizes the difference as around $$\nu_i = \begin{cases} 100\,\text{Hz} & \text{if } \tilde{W}_i = +1, \ 0\,\text{Hz} & \text{if } \tilde{W}_i = 0, \ 50\,\text{Hz} & \text{if } \tilde{W}_i = -1. \end{cases}$$4 accuracy improvement over Poisson rate coding and reports more than $$\nu_i = \begin{cases} 100\,\text{Hz} & \text{if } \tilde{W}_i = +1, \ 0\,\text{Hz} & \text{if } \tilde{W}_i = 0, \ 50\,\text{Hz} & \text{if } \tilde{W}_i = -1. \end{cases}$$5 energy efficiency increase during inference and $$\nu_i = \begin{cases} 100\,\text{Hz} & \text{if } \tilde{W}_i = +1, \ 0\,\text{Hz} & \text{if } \tilde{W}_i = 0, \ 50\,\text{Hz} & \text{if } \tilde{W}_i = -1. \end{cases}$$6 during training relative to traditional deep neural networks, under a compute-only energy model.

The biologically plausible STDP-based encoder reaches $$\nu_i = \begin{cases} 100\,\text{Hz} & \text{if } \tilde{W}_i = +1, \ 0\,\text{Hz} & \text{if } \tilde{W}_i = 0, \ 50\,\text{Hz} & \text{if } \tilde{W}_i = -1. \end{cases}$$7 accuracy on the bydate version of 20 Newsgroups using an encoder with $$\nu_i = \begin{cases} 100\,\text{Hz} & \text{if } \tilde{W}_i = +1, \ 0\,\text{Hz} & \text{if } \tilde{W}_i = 0, \ 50\,\text{Hz} & \text{if } \tilde{W}_i = -1. \end{cases}$$8 neurons and $$\nu_i = \begin{cases} 100\,\text{Hz} & \text{if } \tilde{W}_i = +1, \ 0\,\text{Hz} & \text{if } \tilde{W}_i = 0, \ 50\,\text{Hz} & \text{if } \tilde{W}_i = -1. \end{cases}$$9 pruning (Białas et al., 2020). The reported comparison table places this above KATE at $200$0, CPr-RBM at $200$1, and a Variational Autoencoder at $200$2. The same work reports that pruning $200$3 of weakest weights does not hurt performance much and that the feature space can be reduced from about $200$4 terms to $200$5–$200$6 dimensions while retaining strong accuracy.

In the converted TextCNN setting, directly trained SNNs lag the proposed conversion-plus-fine-tuning pipeline by about $200$7 accuracy on average; pretrained word embeddings improve accuracy by up to $200$8 on average; and the final SNN remains within about $200$9 average drop of the original TextCNN while delivering more than $72$0 lower theoretical energy consumption on average, with up to $72$1 reduction on Waimai (Lv et al., 2024). These results suggest that the quality of the Word2Spike interface is often decisive: the same downstream SNN can behave very differently depending on whether encoding is stochastic, deterministic, pretrained, or directly learned.

5. Integration with spike-driven Transformers and LLMs

Word2Spike becomes more demanding when the downstream model is not a shallow classifier but a spike-native sequence model. The “Spike-driven Transformer” paper, although evaluated on images, is directly relevant because it reformulates token-like processing so that communicated tensors are binary spikes and self-attention is implemented by mask and addition rather than standard multiplication and softmax (Yao et al., 2023). Its Spike-Driven Self-Attention (SDSA) uses spike-form $72$2, $72$3, and $72$4, and the paper reports up to $72$5 lower computation energy than vanilla self-attention together with $72$6 top-1 accuracy on ImageNet-1K. This suggests a blueprint for Word2Spike-style token processing in which dense embeddings are first converted into sparse spike tensors and all subsequent attention becomes spike-native.

SpikeLM extends this logic to general language tasks by arguing that ordinary binary $72$7 spikes are too impoverished for language semantics (Xing et al., 2024). It therefore introduces elastic bi-spiking with direction encoding $72$8, frequency encoding via distribution-aware thresholds, and amplitude encoding $72$9. The paper reports that traditional LIF-based SNNs have about a $-1$0 average performance drop versus ANN baselines, whereas SpikeLM reduces the gap to about $-1$1. In this formulation, Word2Spike is no longer merely input conversion; it becomes a model-wide semantic coding regime.

SpikeDecoder pushes the problem into decoder-only NLP. Its central concern is how to project text into spikes while preserving token identity and positional information in a GPT-style architecture (Beger et al., 10 Jun 2026). The paper compares one-hot encoding, binary position encoding, static positional encoding mapped to spikes, and learned embeddings projected to binary values by a Heaviside function or an initial LIF neuron. Its reported conclusion is that concatenation of learned token embeddings with static positional encoding is the most promising overall compromise, while the proposed SNN-based decoder block reduces theoretical energy consumption by $-1$2 to $-1$3 relative to the ANN baseline. Because the main experiments are character-level, the work also illustrates a recurrent tension in Word2Spike research: whole-word conversion is desirable, but character-level spike interfaces are often easier to stabilize.

6. Limitations, misconceptions, and open problems

A common misconception is that Word2Spike is a settled synonym for Poisson rate coding of word embeddings. The literature does not support that reduction. Some systems rely on stochastic Poisson processes, some on deterministic SAF accumulation, some on TF-IDF-driven probabilistic spike trains, and some on signed or amplitude-aware spike formulations (Kalra et al., 9 Sep 2025, Knipper et al., 2024, Białas et al., 2020, Xing et al., 2024). The representational choice is entangled with downstream objectives: associative recall, classification, robustness, or generative language modeling impose different constraints on fidelity, sparsity, and trainability.

The specific Word2Spike paper is evaluated primarily as a conversion pipeline rather than as a complete associative retrieval system (Kalra et al., 9 Sep 2025). It reports only $-1$4 analogies, uses an untrained spiking transformer as a proof of feasibility, depends on fixed firing-rate choices and thresholding heuristics, and is limited to word-level embeddings rather than sentence or document semantics. These are explicit boundaries on what has been demonstrated.

Other strands expose additional limitations. In the converted TextCNN approach, spike-generating word embeddings are fixed after conversion because the spike trains are random samples and there is no one-to-one deterministic mapping back to a particular real-valued embedding vector (Lv et al., 2024). In the biologically plausible STDP encoder, the optimal division of training data into subsets remains open, inhibition during evaluation must be tuned carefully, the architecture is shallow, and semantic relations between words are not explicitly modeled (Białas et al., 2020). SpikeDecoder notes that full words are more difficult than characters to convert into spikes while retaining information, which is why the main experiments use single characters rather than words (Beger et al., 10 Jun 2026).

The open research agenda is correspondingly broad. Reported next steps include comparing rate coding with temporal coding, varying firing frequencies and observation windows, integrating embedding-to-spike mappings with spiking transformers and liquid state machines, extending evaluation from word-level semantics to more complex language tasks, and determining whether richer spike codes can preserve semantic structure without eroding the energy advantages that motivate neuromorphic NLP in the first place (Kalra et al., 9 Sep 2025, Xing et al., 2024). In that sense, Word2Spike is best understood not as a finished method class but as a still-evolving interface between continuous language representations and spike-driven computation.