Decoding Memory Pipeline

Updated 9 July 2026

Decoding Memory Pipeline (DMP) is a framework that organizes intermediate representations, caches, and retrieval structures for memory-aware decoding.
It optimizes computation by reusing memory states, as demonstrated in FPGA-based LLM inference achieving up to 85% of the theoretical bandwidth limit.
DMP applications extend to self-consistency hallucination detection and EEG decoding, where staged workflows improve inference speed and detection accuracy.

Searching arXiv for the cited DMP-related papers and terminology. arxiv_search query: "Decoding Memory Pipeline (Gao et al., 28 Aug 2025, Li et al., 15 Feb 2025, He et al., 30 Mar 2026, Goldstein et al., 2019)" Decoding Memory Pipeline (DMP) denotes a memory-centric organization of decoding in which intermediate representations, caches, indices, or retrieval structures are arranged so that inference proceeds through an explicit sequence of memory preparation, access, reuse, and application. The expression is used explicitly for a self-consistency framework for hallucination detection in LLMs, and it is also used as a reconstruction or interpretive label for several adjacent systems, including bandwidth-bound LLM decoding on embedded FPGA, sequential EEG decoding of working-memory load, and heterogeneous memory processing for disaggregated LLM inference (Gao et al., 28 Aug 2025, Li et al., 15 Feb 2025, Goldstein et al., 2019, He et al., 30 Mar 2026). This suggests that DMP is better treated as a family of decoding dataflows than as a single canonical algorithm.

1. Scope and principal usages

Across the literature, DMP refers to closely related but not identical constructions. In the most literal sense, it is an inference-time pipeline that stores and reuses prior decoding states to avoid redundant computation. In a broader systems sense, it is a decoding dataflow that treats memory bandwidth, cache layout, and operator scheduling as the primary design variables. In a neuroscience sense, it is a staged workflow for decoding latent cognitive state from temporal neural signals. The common denominator is that decoding quality or efficiency depends on how memory is represented and traversed, rather than on a single isolated predictor.

Context	Pipeline formulation	Representative work
Self-consistency hallucination detection	Selective inference, hard decoding, annealed decoding	(Gao et al., 28 Aug 2025)
Embedded FPGA LLM inference	DDR4 streaming, dequantization, fused attention/MLP, KV writeback	(Li et al., 15 Feb 2025)
EEG working-memory decoding	Acquisition, preprocessing, SVM TOI detection, LSTM-RNN decoding, interpretation	(Goldstein et al., 2019)
Disaggregated LLM inference	Prepare Memory, Compute Relevancy, Retrieval, Apply to Inference	(He et al., 30 Mar 2026)

The term is therefore partly explicit and partly reconstructed. A 2025 hallucination-detection paper names its method DMP directly, whereas several other papers provide the constituent machinery of a decoding memory pipeline without using the exact phrase. This suggests that DMP functions as an organizing abstraction over memory-aware decoding rather than as a settled nomenclature.

2. Memory-bandwidth-limited decoding on embedded FPGA

A hardware-centered DMP is reconstructed most clearly from an embedded FPGA deployment of LLaMA2-7B on the Zynq-based KV260 platform. The system targets a device with 4 GB of 64-bit 2400 Mbps DDR4 memory, where a 4-bit quantized 7B model already occupies about 3.5 GB and decoding is described as purely bandwidth-bound. The model is decoded token by token, layer by layer: the hidden state $h_t$ is projected to $q$ , $k_t$ , and $v_t$ ; RoPE is applied to $q$ and $k_t$ ; quantized $k_t$ and $v_t$ are appended to a per-layer KV cache in DDR4; historical keys and values are read back; attention scores are computed as

$\alpha_i=\mathrm{softmax}\left(\frac{q^\top k_i}{\sqrt{d_{\text{head}}}}\right),\qquad o=\sum_i \alpha_i v_i;$

the attention output is then followed by output projection, residual addition, RMSNorm, and the MLP up, gate, and down projections. Weights are W4, activations are FP16, and the KV cache is 8-bit (Li et al., 15 Feb 2025).

The pipeline is organized around a 4 GB external DDR4 memory, a Memory Control Unit with four 128-bit AXI ports at 300 MHz concatenated into a 512-bit stream, on-chip dequantization and buffering, a Vector Processing Unit with 128 FP16 multipliers and an adder tree, and a Scalar Processing Unit that implements RoPE, RMSNorm,

$\mathrm{RMS}(a)=\sqrt{\frac{1}{n}\sum_i a_i^2},\qquad \bar a_i=\frac{a_i}{\mathrm{RMS}(a)},$

softmax via an online stable three-pass algorithm, SiLU,

$q$ 0

and two-pass linear quantization for KV. Quantized K/V and packed scale-zero metadata are written back to DDR, while all intermediate hidden states remain on chip during decode.

The resulting DMP is explicitly memory-centric. The system is developed in a bare-metal environment so that memory capacity is reserved for weights and KV cache rather than for an operating system. It uses a customized dataflow with operator fusion and a bus-width-aligned data arrangement in which weights, zero points, and scales are interleaved to maximize sequential burst efficiency. A 512-bit DDR transaction can carry 64 4-bit weights, 64 4-bit zero points, and 16 16-bit scales, and KV scale-zero metadata are packed into 32-bit units and accumulated into full-width bursts. The design reserves 93.3% of memory capacity for weights and KV cache, reaches a measured decoding speed of about 4.9 token/s against a theoretical 5.8 token/s, and thus attains 84.5% bandwidth utilization; the abstract reports this as around 5 token/s, 93.3% capacity utilization, and 85% of the theoretical bandwidth limit (Li et al., 15 Feb 2025).

3. Explicit DMP in self-consistency hallucination detection

In its explicit LLM usage, DMP is an inference-side framework for accelerating self-consistency hallucination detection. Self-consistency methods generate multiple responses

$q$ 1

for the same prompt, then score agreement by length-normalized entropy, lexical similarity, semantic entropy, SelfCheckGPT, EigenScore, or related measures. Their main computational cost is repeated autoregressive decoding. The DMP formulation begins from the observation that these generations contain shared prefixes and many non-exact-answer tokens whose variation contributes minimally to semantic content. It therefore introduces a decoding memory

$q$ 2

for each completed response, storing token sequences, logits, full-sequence KV cache, and final-layer hidden embeddings (Gao et al., 28 Aug 2025).

The first mechanism is selective inference. If the current partial response $q$ 3 matches the prefix of a stored response $q$ 4, the next token, logits, and KV states are reused:

$q$ 5

The second mechanism is hard decoding: when the cached distribution is confident, with threshold $q$ 6 in the reported configuration, the cached argmax token is deterministically reused. The third mechanism is annealed decoding. Token importance is estimated by cosine similarity between the prompt embedding $q$ 7 and token embedding $q$ 8,

$q$ 9

and logits for non-important tokens are rescaled by $k_t$ 0,

$k_t$ 1

which is equivalent to lowering the effective temperature from $k_t$ 2 to $k_t$ 3 for those tokens. Short answers with fewer than 10 tokens are excluded from annealing.

This DMP reduces the effective number of forward passes from $k_t$ 4 to $k_t$ 5, where $k_t$ 6 is the reuse ratio, yielding an ideal speedup of $k_t$ 7. Empirically, reuse ratios of roughly 50–66% are reported across models and datasets, with observed speedups from 1.9× to 3×. For LLaMA-2-7B-Chat on TriviaQA, the reuse ratio is 66.8% and runtime drops from 1778 s to 627 s, a 2.8× speedup, while mean AUROC over baselines changes from 74.09 to 74.11. Across eight main trials, the mean AUROC with DMP exceeds the baseline in five, and the maximum AUROC drop over all baselines and datasets is 2.9% (Gao et al., 28 Aug 2025).

4. Sequential neural decoding of working memory load

A neurodecoding DMP is reconstructed from an EEG study of working-memory load that formalizes a full pipeline from task design to interpretation. The stages are: structured task and data acquisition with a Sternberg verbal working-memory task; preprocessing into cleaned and standardized current source density time series; initial time-point decoding with a linear SVM to identify time periods of interest; sequential modeling with a many-to-one LSTM-RNN; training and evaluation by cross-entropy and Adam; and interpretation by topographical gate-weight maps. Each sample is a sequence $k_t$ 8 of 128-dimensional EEG vectors, and the central comparison is between ordered sequences, which preserve temporal structure, and temporally shuffled sequences, which preserve spatial patterns while destroying order (Goldstein et al., 2019).

The preprocessing stack is itself memory-structuring. High-density 128-channel BioSemi ActiveTwo EEG is filtered from 0.1 to 30 Hz with zero-phase FIR filters, downsampled from 1000 Hz to 256 Hz, average-referenced, epoched from −1 s to 7 s relative to cue onset, artifact-rejected, cleaned with ICA in EEGLAB, converted to reference-free CSD via a 2D surface Laplacian, smoothed with a Gaussian kernel of length 5 samples, and standardized across electrodes. The LSTM receives 128-channel inputs, uses 50 cells, and outputs a binary softmax for Load 2 versus Load 6. The core equations are

$k_t$ 9

$v_t$ 0

$v_t$ 1

followed by a softmax classifier on $v_t$ 2.

The reported results establish a temporal-memory interpretation of decoding. Accuracy increases with sequence length in both ordered and shuffled conditions, but more rapidly for ordered sequences. Ordered sequences outperform shuffled sequences in encoding, retention, activity-silent, and retrieval periods, with the difference growing with window length. Plateaued accuracies are highest during retention, followed by encoding and retrieval, and lowest but still above chance during the activity-silent period. During activity-silent intervals, SVM decoding at single time points falls to chance, whereas LSTM decoding rises above chance on longer windows; however, ordered and shuffled accuracies are not significantly different there, suggesting that the decodable signal in that phase resides mainly in subtle temporally static patterns rather than in strong sequential structure. Topographical decoding weights emphasize frontal, temporal, and some parietal regions, and the forget gate shows larger absolute weights in ordered than shuffled data, whereas the input gate shows the opposite pattern (Goldstein et al., 2019).

5. Generalized memory-processing formulation

A generalized formulation of DMP in LLM inference appears in work on disaggregated memory processing. There, inference at time $v_t$ 3 is written as

$v_t$ 4

where $v_t$ 5 is a memory generator and $v_t$ 6 is a memory processor. The processor is decomposed into four steps: Prepare Memory, Compute Relevancy, Retrieval, and Apply to Inference. Prepare Memory converts raw memory into an index or structured representation; Compute Relevancy scores memory entries against the current token or query; Retrieval selects or aggregates the relevant subset; Apply to Inference integrates the result into decoding. The framework is used to unify sparse attention, retrieval-augmented generation, synthesized memory, and compressed contextual memory (He et al., 30 Mar 2026).

This four-step abstraction makes explicit what the more specialized DMPs are doing. In the FPGA decoder, Prepare Memory corresponds to weight arrangement, on-chip dequantization, and KV packing; Compute Relevancy and Retrieval correspond to sequential access to KV8 and head-wise attention; Apply to Inference corresponds to the fused attention and MLP path. In self-consistency hallucination detection, the same roles are played by construction of the decoding memory, prefix matching, selective reuse, and downstream hallucination scoring. In EEG decoding, preprocessing and TOI selection prepare memory-like representations, the LSTM gates compute temporal relevancy, the hidden state retrieves prior sequence information, and the final softmax applies that information to classification. This suggests that DMP is best defined by a memory-processing pattern rather than by a fixed model class.

The systems implications are similar across these settings. The disaggregated-inference study reports that memory processing accounts for 22%–97% of total latency, with Compute Relevancy and Retrieval often exhibiting arithmetic intensity of roughly 1–10 and irregular access patterns, whereas Apply to Inference is typically denser and more compute-bound. On an AMD MI210 GPU and an Alveo U55C FPGA, offloading sparse, irregular, memory-bounded operations to FPGA yields end-to-end speedups of 1.04×–2.2× and energy reductions of 1.11×–4.7× across multiple inference optimizations (He et al., 30 Mar 2026).

The expression DMP is not unique to decoding. It should not be conflated with the “Dual Memory Pathway” architecture in neuromorphic spiking networks, which couples a fast spiking pathway with a slow low-dimensional state-space memory and reports more than a 4× increase in throughput and over a 5× improvement in energy efficiency relative to prior implementations (Sun et al., 8 Dec 2025). Nor should it be conflated with “Data Memory-dependent Prefetchers,” microarchitectural prefetchers that inspect memory contents and motivate compiler defenses such as SplittingSecrets (Sharma et al., 18 Jan 2026). A related but differently named framework is Parametric Memory Decoding, which reframes zero-shot LoRA routing as decoding activations over external parametric memory and instantiates PMDRouter via scale-normalized response energy after a single base-model prefill (Ji et al., 5 Jul 2026).

A further neighboring line of work is pipeline-based speculative decoding. PipeDec integrates a draft model into a 14-stage parallel pipeline, manages speculative branches with a dynamic prediction tree, and reports decoding-phase speedups of 4.46×–7.79× over traditional pipeline parallelism and 2.2×–2.69× over baseline tree-based speculative decoding, but it does not use the DMP label (Yin et al., 5 Apr 2025). This suggests that “Decoding Memory Pipeline” currently denotes a conceptual region of research rather than a closed taxonomy.

The principal misconception is therefore terminological: DMP is not a single universally standardized method. In the cited literature it names one specific hallucination-detection pipeline, while in other settings it is an accurate reconstruction of memory-aware decoding dataflows. The stable technical content lies in the design commitments shared across these works: explicit state reuse, structured memory layouts, staged relevancy computation, on-chip or in-cache retention of intermediate state where possible, and scheduling policies that treat memory traffic as a first-class bottleneck.