---
title: Scratchpad Patching in Byte-level Modeling
url: https://www.emergentmind.com/topics/scratchpad-patching-sp
type: topic
---

# Scratchpad Patching in Byte-level Modeling

Searching arXiv for the cited papers and closely related terminology.
Scratchpad Patching (SP) is a mechanism for tokenizer-free, byte-level, patch-based language models that inserts transient scratchpads inside each patch to aggregate the bytes seen so far and refresh patch-level context for subsequent predictions. It was introduced to address the quality degradation caused by large patch sizes in models that group contiguous byte spans into patches for efficiency. In that setting, SP is presented as a way to decouple within-patch compute from the committed patch size, so that inference-time compute can be adjusted post hoc while preserving the reduced KV-cache footprint associated with patch-based processing [2605.09630].

## 1. Problem setting: byte-level patching and patch lag

In a tokenizer-free model, computation proceeds directly on a UTF-8 byte stream \(b_1, b_2, \ldots, b_n \in \{0 \ldots 255\}\) plus a few sentinels. Because \(N\) is large, patch-based methods group contiguous byte windows \([s_\ell,e_\ell]\) into \(L\) patches, each summarized to a single patch representation \(z_\ell\) before passing through the expensive Transformer trunk \(\mathcal{M}\). A typical byte-patch model has the pipeline \(\text{Encoder } \mathcal{E} \rightarrow \text{Patchifier } \mathcal{P} \rightarrow \text{Trunk } \mathcal{M} \rightarrow \text{Unpatchifier } \mathcal{U} \rightarrow \text{Decoder } \mathcal{D}\) [2605.09630].

This design creates a trade-off. Larger patches imply fewer trunk steps, lower compute, and a smaller KV-cache. However, large patches also introduce “patch lag”: until a patch is fully observed, byte predictions within it must rely on a stale representation from the previous patch in order to preserve causality. In the standard non-SP unpatchifier, for patch \(\ell\) covering bytes \(n \in [s_\ell,e_\ell]\),

\[
u_n \;=\;
\begin{cases}
\widetilde z_{\,\ell-1} + x_n, & n \ne e_\ell,\\
\widetilde z_{\,\ell} + x_n, & n = e_\ell,
\end{cases}
\]

where \(x_n=\mathcal{E}(b_n)\), \(z_\ell=\mathrm{Aggregate}(x_{s_\ell:e_\ell})\), and \(\widetilde z_\ell=\mathcal{M}(z_\ell)\). For non-final bytes, the model therefore conditions on \(\widetilde z_{\ell-1}\) rather than an updated summary of the current patch. If each patch is on average \(P\) bytes, the lag can be as large as \(P\) steps [2605.09630].

The central motivation for SP is thus not generic scratchpad reasoning, but a specific systems-level problem in byte-level autoregressive modeling: retaining most of the efficiency benefit of patch compression without forcing intra-patch prediction to depend on stale context.

## 2. Core mechanism: transient scratchpads inside a patch

SP inserts transient “scratchpad” states inside each patch so that non-final bytes can condition on a more current representation of their own patch. For patch \(\ell\) spanning \([s_\ell,e_\ell]\), binary triggers \(p_n \in \{0,1\}\) are defined for \(n \in [s_\ell,e_\ell]\), and the number of scratchpads is

\[
T_\ell = \sum_{n=s_\ell}^{e_\ell} p_n.
\]

The construction reserves \(z_\ell^0 \equiv z_{\ell-1}\), and for each \(t \in \{1 \ldots T_\ell\}\), \(z_\ell^t = \mathrm{Aggregate}(x_{s_\ell:n_t})\), where \(n_t\) is the \(t\)-th position with \(p_n=1\). Each \(z_\ell^t\) is passed through the trunk to produce \(\widetilde z_\ell^t=\mathcal{M}(z_\ell^t)\). The unpatchifier then becomes

\[
u_n \;=\;
\begin{cases}
\widetilde z_{\ell}^{\,t(n)} + x_n, & n \neq e_\ell,\\
\widetilde z_{\ell} + x_n, & n = e_\ell,
\end{cases}
\]

where \(t(n)=\sum_{j=s_\ell}^n p_j\) is the index of the most recent scratchpad before or at \(n\). Thus every byte except the final byte conditions on some \(\widetilde z_\ell^t\) from its own patch, rather than always using \(\widetilde z_{\ell-1}\) [2605.09630].

The scratchpads are transient rather than committed. Only the final patch representation \(z_\ell\) is committed to the cache for future patches. This distinction is structurally important: SP increases within-patch compute adaptively, but does not change the asymptotic cache footprint of the committed patch sequence.

A useful way to characterize the mechanism is that committed patch size controls inter-patch memory, whereas scratchpads control intra-patch refresh. This interpretation follows directly from the fact that scratchpads are recomputed within a patch, used to improve subsequent byte prediction inside that patch, and then dropped from the KV cache at inference time [2605.09630].

## 3. Triggering, masking, and inference procedure

SP does not recompute at every byte. Instead, it uses an auxiliary next-byte entropy \(H_n\) from a lightweight LM head over \(x_{\le n}\):

\[
H_n \;=\; -\sum_{b\in V} p(b\mid x_{\le n})\log p(b\mid x_{\le n}),
\]

and sets

\[
p_n \;=\; \mathbf{1}[\,H_n>\tau_{\mathrm{SP}}\,].
\]

The threshold \(\tau_{\mathrm{SP}}\) is tuned per patchifier. Common defaults are \(\tau_{\mathrm{SP}}\approx 1.5\) nats for fixed/SpaceByte, \(\tau_{\mathrm{SP}}\approx 1.0\) for entropy-based patching, and \(\tau_{\mathrm{SP}}\approx 2.5\) for H-Net in order to avoid redundant near-boundary updates [2605.09630].

Training is parallelized by unrolling all patch and scratchpad states as a single sequence,
\[
[z_0, z_1^1\ldots z_1^{T_1}, z_1, z_2^1\ldots z_2^{T_2}, z_2, \ldots],
\]
while masking self-attention so that each scratchpad or patch \(\ell\) attends only to \(z_0\ldots z_{\ell-1}\), and no attention flows between scratchpads of the same patch. This allows gradients to be computed in one pass. At inference, scratchpads are omitted from the KV cache because they are never keys for later patches [2605.09630].

The implementation notes emphasize several constraints. Scratchpad states must share the patch’s positional index and must not be attended to by later patches. Gradients from the entropy head back into \(\mathcal{E}\) are stopped to stabilize representation learning. An entropy head with two extra Transformer layers on top of \(\mathcal{E}\) works well, and additional layers provide little benefit. The threshold \(\tau_{\mathrm{SP}}\) is described as critical: values that are too low waste compute, while values that are too high produce too few scratchpads. SP is intended to integrate with any patchifier, provided that \(p_n=0\) on real patch boundaries to avoid double-counting [2605.09630].

A recurring misunderstanding is that a scratchpad mechanism should be dense to be effective. The reported ablations do not support that view. Dense updates, corresponding to \(\tau_{\mathrm{SP}}\approx 0\), yield diminishing returns or even degrade BPB on natural text; the more effective regime is selective triggering at information-dense positions [2605.09630].

## 4. Compute, memory, and empirical performance

The complexity profile is central to SP. In a non-SP model, \(L\) patches imply \(L\) trunk states in cache and memory \(O(L)\), with compute approximately \(O(L \cdot C_{\mathrm{dir}})\), where \(C_{\mathrm{dir}}\) is the cost per trunk token. With SP, there are still only \(L\) committed representations in cache, but patch \(\ell\) requires \(T_\ell+1\) trunk calls instead of one. Total inference trunk FLOPs are therefore proportional to \(\sum_\ell (1+T_\ell)\) rather than \(\sum_\ell 1\). In practice, entropy triggering keeps \(T_\ell\) much smaller than patch size on average, so the reported overhead is only \(0.5\)–\(1.0\times\) extra compute while preserving the \(O(L)\) cache [2605.09630].

Empirically, SP shifts the BPB–compression Pareto frontier downward across four patchifier families: Fixed-size \(p=2,4,8,16\), SpaceByte, Entropy, and H-Net. At \(p=16\), SP is reported to recover almost all of the BPB loss of vanilla patching, bringing SP-Fixed \((p=16)\) within \(1\%\) of the byte-level baseline [2605.09630].

The reported downstream results on natural language understanding and code generation are summarized below.

| Model | Key efficiency figure | Reported task result |
|---|---:|---:|
| Byte-level | SeqRed \(1.0\times\), FLOPsRed \(1.0\times\) | AvgAcc \(54.1\%\) |
| Tokenizer-based | SeqRed \(3.7\times\), FLOPsRed \(4.1\times\) | AvgAcc \(55.9\%\) |
| Fixed \((p=16)\) | SeqRed \(16.0\times\), FLOPsRed \(5.7\times\) | AvgAcc \(48.0\%\) |
| Fixed \((p=16)+\)SP | SeqRed \(16.0\times\), FLOPsRed \(2.9\times\) | AvgAcc \(54.2\%\) |

| Model | MBPP P@1 | HE P@1 |
|---|---:|---:|
| Byte-level | \(26.3\%\) | \(15.5\%\) |
| Tokenizer-based | \(23.3\%\) | \(13.3\%\) |
| Fixed \((p=16)\) | \(18.2\%\) | \(10.5\%\) |
| Fixed \((p=16)+\)SP | \(27.5\%\) | \(14.8\%\) |

For code generation, the same \(p=16\) SP configuration preserves a \(16\times\) KV-cache reduction while reducing inference FLOPs by \(3.7\times\) on MBPP and \(4.5\times\) on HumanEval. The improvement over vanilla patching at \(p=16\) is reported as approximately \(9\) percentage points on MBPP, from \(18.2\) to \(27.5\), and approximately \(4\) percentage points on HumanEval, from \(10.5\) to \(14.8\) [2605.09630].

These results are significant in the narrow technical sense established by the paper: they indicate that the quality loss traditionally attributed to large patch sizes is not purely a consequence of compression ratio, but is closely tied to stale within-patch context. SP addresses that specific failure mode without forfeiting the cache savings of patch-level commitment.

## 5. Related meanings of “scratchpad patching” in reasoning and robotics

The phrase “scratchpad patching” also appears in two distinct but related 2026 contexts: intervention on reasoning traces and update rules for language scratchpads in vision-language-action systems.

In "Do Models Read What They Write? Causal Registers in Scratchpad Reasoning" [2606.29522], the setting is a controlled state-tracking problem with known transition rule
\[
s_{t+1}=f_G(s_t,a_t)=(v_t \oplus u(a_t),\, p_t \oplus \delta_G(v_t,a_t)),
\]
where \(v_t \in \{00,01,10,11\}\), \(p_t \in \{0,1\}\), and \(a_t \in \{Q,W,T,Z\}\). There, a model fine-tuned to emit intermediate states writes each \(s_t\) explicitly in a visible scratchpad. The intervention protocol edits the internal residual-stream representation of the printed current phase token at a chosen layer while leaving the visible scratchpad text fixed. The resulting “patch” tests whether later computation follows the edited latent state rather than the original visible text. In Qwen2.5-Coder-7B, the running-state model achieves edited-branch agreement \(R \approx 0.80\) on the \(Q_8\) task and \(R \approx 0.91\) on the \(D_8\) task, while pretrained and final-answer-only controls remain near \(R \approx 0.50\), and random or orthogonal patches yield \(R \approx 0.02\) [2606.29522].

In "Notes-to-Self: Scratchpad Augmented VLAs for Memory Dependent Manipulation Tasks" [2602.21013], scratchpad patching refers to updating an external sequence of text tokens \(S_t=[d_1,d_2,\ldots,d_k]\) that is prepended or interleaved with the current image and language goal at each timestep. The policy outputs an action \(a_t\) and a language description \(d_t\), and the scratchpad update is governed by a gating bit \(\delta_t\): if \(d_t\) includes an update trigger such as \(\langle\mathrm{done}\rangle\), then \(S_{t+1}=S_t \cup \{d_t\}\); otherwise the scratchpad is left unchanged. In the reported ClevrSkills-Mem experiments, average success across five tasks rises from approximately \(10\%\) to approximately \(58\%\) for T-VLA with SP, and from approximately \(60\%\) to approximately \(71\%\) for R-VLA with SP [2602.21013].

These usages are not identical to the byte-level SP mechanism. In the byte-level paper, scratchpads are transient internal summaries inside a patch; in the causal-reasoning paper, patching is an activation intervention on a written intermediate state; in the VLA paper, patching is an explicit update rule for an external text memory. What unifies them is the attempt to make intermediate state representations operational rather than merely decorative. This suggests that “scratchpad” functions as a broader design motif across multiple subfields, while “Scratchpad Patching (SP)” in the strict acronymized sense refers specifically to the byte-level patch-based mechanism of [2605.09630].

## 6. Significance, limitations, and interpretive scope

The principal technical significance of SP in byte-level language modeling is that it reframes the compute-quality trade-off of patch-based models. Larger committed patches no longer imply a single fixed amount of within-patch computation, because scratchpads permit selective refresh at high-entropy positions. The paper therefore describes SP as decoupling compute from patch size, rather than replacing patching itself [2605.09630].

The ablations also identify clear limits. Entropy-based triggers outperform fixed-stride and whitespace heuristics in BPB versus FLOPs for a Fixed \(p=8\) base. Dense updates are not uniformly beneficial. Under H-Net, scratchpad triggers often land one byte before H-Net boundaries, causing redundant near-boundary computations, which explains smaller gains under strict FLOPs matching. These findings indicate that SP is not a universal monotonic improvement; its behavior depends on interaction with the chosen patchifier and on threshold calibration [2605.09630].

The related reasoning and robotics papers sharpen a broader conceptual point. In the reasoning study, “causal legibility” is distinguished from “plausible legibility”: it is not enough for a model to write a plausible chain of thought if later computation does not use the written variables [2606.29522]. In the VLA setting, the scratchpad serves as a persistent textual memory for object positions, plans, and progress toward subgoals, and improvements are largest on memory-dependent long-horizon tasks [2602.21013]. A plausible implication is that contemporary scratchpad research is converging on a shared criterion: intermediate states become scientifically useful when they alter downstream computation in predictable ways.

Within that broader landscape, Scratchpad Patching in the sense of [2605.09630] occupies a specific systems and architecture niche. It is a method for improving byte-level autoregressive modeling under sequence compression by repairing stale intra-patch context, using transient, entropy-triggered summaries that are computationally active but not persistently cached.

Source: https://www.emergentmind.com/topics/scratchpad-patching-sp