Scratchpad Patching in Byte-level Modeling

• Scratchpad Patching (SP) is a byte-level mechanism that decouples compute from patch size by inserting transient scratchpads to update stale intra-patch context.
• It uses entropy-triggered binary signals to selectively refresh patch representations, thereby mitigating patch lag and improving prediction accuracy.
• Empirical results show that SP nearly recovers the quality loss of large patches while maintaining a reduced KV-cache footprint and lowering inference FLOPs.

Searching arXiv for the cited papers and closely related terminology. Scratchpad Patching (SP) is a mechanism for tokenizer-free, byte-level, patch-based LLMs that inserts transient scratchpads inside each patch to aggregate the bytes seen so far and refresh patch-level context for subsequent predictions. It was introduced to address the quality degradation caused by large patch sizes in models that group contiguous byte spans into patches for efficiency. In that setting, SP is presented as a way to decouple within-patch compute from the committed patch size, so that inference-time compute can be adjusted post hoc while preserving the reduced KV-cache footprint associated with patch-based processing (Zheng et al., 10 May 2026).

1. Problem setting: byte-level patching and patch lag

In a tokenizer-free model, computation proceeds directly on a UTF-8 byte stream $b_1, b_2, \ldots, b_n \in \{0 \ldots 255\}$ plus a few sentinels. Because $N$ is large, patch-based methods group contiguous byte windows $[s_\ell,e_\ell]$ into $L$ patches, each summarized to a single patch representation $z_\ell$ before passing through the expensive Transformer trunk $\mathcal{M}$. A typical byte-patch model has the pipeline $$\text{Encoder } \mathcal{E} \rightarrow \text{Patchifier } \mathcal{P} \rightarrow \text{Trunk } \mathcal{M} \rightarrow \text{Unpatchifier } \mathcal{U} \rightarrow \text{Decoder } \mathcal{D}$$ (Zheng et al., 10 May 2026).

This design creates a trade-off. Larger patches imply fewer trunk steps, lower compute, and a smaller KV-cache. However, large patches also introduce “patch lag”: until a patch is fully observed, byte predictions within it must rely on a stale representation from the previous patch in order to preserve causality. In the standard non-SP unpatchifier, for patch $\ell$ covering bytes $n \in [s_\ell,e_\ell]$,

$$u_n \;=\; \begin{cases} \widetilde z_{\,\ell-1} + x_n, & n \ne e_\ell,\ \widetilde z_{\,\ell} + x_n, & n = e_\ell, \end{cases}$$

where $N$0, $N$1, and $N$2. For non-final bytes, the model therefore conditions on $N$3 rather than an updated summary of the current patch. If each patch is on average $N$4 bytes, the lag can be as large as $N$5 steps (Zheng et al., 10 May 2026).

The central motivation for SP is thus not generic scratchpad reasoning, but a specific systems-level problem in byte-level autoregressive modeling: retaining most of the efficiency benefit of patch compression without forcing intra-patch prediction to depend on stale context.

2. Core mechanism: transient scratchpads inside a patch

SP inserts transient “scratchpad” states inside each patch so that non-final bytes can condition on a more current representation of their own patch. For patch $N$6 spanning $N$7, binary triggers $N$8 are defined for $N$9, and the number of scratchpads is

$[s_\ell,e_\ell]$0

The construction reserves $[s_\ell,e_\ell]$1, and for each $[s_\ell,e_\ell]$2, $[s_\ell,e_\ell]$3, where $[s_\ell,e_\ell]$4 is the $[s_\ell,e_\ell]$5-th position with $[s_\ell,e_\ell]$6. Each $[s_\ell,e_\ell]$7 is passed through the trunk to produce $[s_\ell,e_\ell]$8. The unpatchifier then becomes

$[s_\ell,e_\ell]$9

where $L$0 is the index of the most recent scratchpad before or at $L$1. Thus every byte except the final byte conditions on some $L$2 from its own patch, rather than always using $L$3 (Zheng et al., 10 May 2026).

The scratchpads are transient rather than committed. Only the final patch representation $L$4 is committed to the cache for future patches. This distinction is structurally important: SP increases within-patch compute adaptively, but does not change the asymptotic cache footprint of the committed patch sequence.

A useful way to characterize the mechanism is that committed patch size controls inter-patch memory, whereas scratchpads control intra-patch refresh. This interpretation follows directly from the fact that scratchpads are recomputed within a patch, used to improve subsequent byte prediction inside that patch, and then dropped from the KV cache at inference time (Zheng et al., 10 May 2026).

3. Triggering, masking, and inference procedure

SP does not recompute at every byte. Instead, it uses an auxiliary next-byte entropy $L$5 from a lightweight LM head over $L$6:

$L$7

and sets

$L$8

The threshold $L$9 is tuned per patchifier. Common defaults are $z_\ell$0 nats for fixed/SpaceByte, $z_\ell$1 for entropy-based patching, and $z_\ell$2 for H-Net in order to avoid redundant near-boundary updates (Zheng et al., 10 May 2026).

Training is parallelized by unrolling all patch and scratchpad states as a single sequence,

$z_\ell$3

while masking self-attention so that each scratchpad or patch $z_\ell$4 attends only to $z_\ell$5, and no attention flows between scratchpads of the same patch. This allows gradients to be computed in one pass. At inference, scratchpads are omitted from the KV cache because they are never keys for later patches (Zheng et al., 10 May 2026).

The implementation notes emphasize several constraints. Scratchpad states must share the patch’s positional index and must not be attended to by later patches. Gradients from the entropy head back into $z_\ell$6 are stopped to stabilize representation learning. An entropy head with two extra Transformer layers on top of $z_\ell$7 works well, and additional layers provide little benefit. The threshold $z_\ell$8 is described as critical: values that are too low waste compute, while values that are too high produce too few scratchpads. SP is intended to integrate with any patchifier, provided that $z_\ell$9 on real patch boundaries to avoid double-counting (Zheng et al., 10 May 2026).

A recurring misunderstanding is that a scratchpad mechanism should be dense to be effective. The reported ablations do not support that view. Dense updates, corresponding to $\mathcal{M}$0, yield diminishing returns or even degrade BPB on natural text; the more effective regime is selective triggering at information-dense positions (Zheng et al., 10 May 2026).

4. Compute, memory, and empirical performance

The complexity profile is central to SP. In a non-SP model, $\mathcal{M}$1 patches imply $\mathcal{M}$2 trunk states in cache and memory $\mathcal{M}$3, with compute approximately $\mathcal{M}$4, where $\mathcal{M}$5 is the cost per trunk token. With SP, there are still only $\mathcal{M}$6 committed representations in cache, but patch $\mathcal{M}$7 requires $\mathcal{M}$8 trunk calls instead of one. Total inference trunk FLOPs are therefore proportional to $\mathcal{M}$9 rather than $$\text{Encoder } \mathcal{E} \rightarrow \text{Patchifier } \mathcal{P} \rightarrow \text{Trunk } \mathcal{M} \rightarrow \text{Unpatchifier } \mathcal{U} \rightarrow \text{Decoder } \mathcal{D}$$0. In practice, entropy triggering keeps $$\text{Encoder } \mathcal{E} \rightarrow \text{Patchifier } \mathcal{P} \rightarrow \text{Trunk } \mathcal{M} \rightarrow \text{Unpatchifier } \mathcal{U} \rightarrow \text{Decoder } \mathcal{D}$$1 much smaller than patch size on average, so the reported overhead is only $$\text{Encoder } \mathcal{E} \rightarrow \text{Patchifier } \mathcal{P} \rightarrow \text{Trunk } \mathcal{M} \rightarrow \text{Unpatchifier } \mathcal{U} \rightarrow \text{Decoder } \mathcal{D}$$2–$$\text{Encoder } \mathcal{E} \rightarrow \text{Patchifier } \mathcal{P} \rightarrow \text{Trunk } \mathcal{M} \rightarrow \text{Unpatchifier } \mathcal{U} \rightarrow \text{Decoder } \mathcal{D}$$3 extra compute while preserving the $$\text{Encoder } \mathcal{E} \rightarrow \text{Patchifier } \mathcal{P} \rightarrow \text{Trunk } \mathcal{M} \rightarrow \text{Unpatchifier } \mathcal{U} \rightarrow \text{Decoder } \mathcal{D}$$4 cache (Zheng et al., 10 May 2026).

Empirically, SP shifts the BPB–compression Pareto frontier downward across four patchifier families: Fixed-size $$\text{Encoder } \mathcal{E} \rightarrow \text{Patchifier } \mathcal{P} \rightarrow \text{Trunk } \mathcal{M} \rightarrow \text{Unpatchifier } \mathcal{U} \rightarrow \text{Decoder } \mathcal{D}$$5, SpaceByte, Entropy, and H-Net. At $$\text{Encoder } \mathcal{E} \rightarrow \text{Patchifier } \mathcal{P} \rightarrow \text{Trunk } \mathcal{M} \rightarrow \text{Unpatchifier } \mathcal{U} \rightarrow \text{Decoder } \mathcal{D}$$6, SP is reported to recover almost all of the BPB loss of vanilla patching, bringing SP-Fixed $$\text{Encoder } \mathcal{E} \rightarrow \text{Patchifier } \mathcal{P} \rightarrow \text{Trunk } \mathcal{M} \rightarrow \text{Unpatchifier } \mathcal{U} \rightarrow \text{Decoder } \mathcal{D}$$7 within $$\text{Encoder } \mathcal{E} \rightarrow \text{Patchifier } \mathcal{P} \rightarrow \text{Trunk } \mathcal{M} \rightarrow \text{Unpatchifier } \mathcal{U} \rightarrow \text{Decoder } \mathcal{D}$$8 of the byte-level baseline (Zheng et al., 10 May 2026).

The reported downstream results on natural language understanding and code generation are summarized below.

Model	Key efficiency figure	Reported task result
Byte-level	SeqRed $$\text{Encoder } \mathcal{E} \rightarrow \text{Patchifier } \mathcal{P} \rightarrow \text{Trunk } \mathcal{M} \rightarrow \text{Unpatchifier } \mathcal{U} \rightarrow \text{Decoder } \mathcal{D}$$9, FLOPsRed $\ell$0	AvgAcc $\ell$1
Tokenizer-based	SeqRed $\ell$2, FLOPsRed $\ell$3	AvgAcc $\ell$4
Fixed $\ell$5	SeqRed $\ell$6, FLOPsRed $\ell$7	AvgAcc $\ell$8
Fixed $\ell$9SP	SeqRed $n \in [s_\ell,e_\ell]$0, FLOPsRed $n \in [s_\ell,e_\ell]$1	AvgAcc $n \in [s_\ell,e_\ell]$2

Model	MBPP P@1	HE P@1
Byte-level	$n \in [s_\ell,e_\ell]$3	$n \in [s_\ell,e_\ell]$4
Tokenizer-based	$n \in [s_\ell,e_\ell]$5	$n \in [s_\ell,e_\ell]$6
Fixed $n \in [s_\ell,e_\ell]$7	$n \in [s_\ell,e_\ell]$8	$n \in [s_\ell,e_\ell]$9
Fixed $$u_n \;=\; \begin{cases} \widetilde z_{\,\ell-1} + x_n, & n \ne e_\ell,\ \widetilde z_{\,\ell} + x_n, & n = e_\ell, \end{cases}$$0SP	$$u_n \;=\; \begin{cases} \widetilde z_{\,\ell-1} + x_n, & n \ne e_\ell,\ \widetilde z_{\,\ell} + x_n, & n = e_\ell, \end{cases}$$1	$$u_n \;=\; \begin{cases} \widetilde z_{\,\ell-1} + x_n, & n \ne e_\ell,\ \widetilde z_{\,\ell} + x_n, & n = e_\ell, \end{cases}$$2

For code generation, the same $$u_n \;=\; \begin{cases} \widetilde z_{\,\ell-1} + x_n, & n \ne e_\ell,\ \widetilde z_{\,\ell} + x_n, & n = e_\ell, \end{cases}$$3 SP configuration preserves a $$u_n \;=\; \begin{cases} \widetilde z_{\,\ell-1} + x_n, & n \ne e_\ell,\ \widetilde z_{\,\ell} + x_n, & n = e_\ell, \end{cases}$$4 KV-cache reduction while reducing inference FLOPs by $$u_n \;=\; \begin{cases} \widetilde z_{\,\ell-1} + x_n, & n \ne e_\ell,\ \widetilde z_{\,\ell} + x_n, & n = e_\ell, \end{cases}$$5 on MBPP and $$u_n \;=\; \begin{cases} \widetilde z_{\,\ell-1} + x_n, & n \ne e_\ell,\ \widetilde z_{\,\ell} + x_n, & n = e_\ell, \end{cases}$$6 on HumanEval. The improvement over vanilla patching at $$u_n \;=\; \begin{cases} \widetilde z_{\,\ell-1} + x_n, & n \ne e_\ell,\ \widetilde z_{\,\ell} + x_n, & n = e_\ell, \end{cases}$$7 is reported as approximately $$u_n \;=\; \begin{cases} \widetilde z_{\,\ell-1} + x_n, & n \ne e_\ell,\ \widetilde z_{\,\ell} + x_n, & n = e_\ell, \end{cases}$$8 percentage points on MBPP, from $$u_n \;=\; \begin{cases} \widetilde z_{\,\ell-1} + x_n, & n \ne e_\ell,\ \widetilde z_{\,\ell} + x_n, & n = e_\ell, \end{cases}$$9 to $N$00, and approximately $N$01 percentage points on HumanEval, from $N$02 to $N$03 (Zheng et al., 10 May 2026).

These results are significant in the narrow technical sense established by the paper: they indicate that the quality loss traditionally attributed to large patch sizes is not purely a consequence of compression ratio, but is closely tied to stale within-patch context. SP addresses that specific failure mode without forfeiting the cache savings of patch-level commitment.

5. Related meanings of “scratchpad patching” in reasoning and robotics

The phrase “scratchpad patching” also appears in two distinct but related 2026 contexts: intervention on reasoning traces and update rules for language scratchpads in vision-language-action systems.

In "Do Models Read What They Write? Causal Registers in Scratchpad Reasoning" (Shih et al., 28 Jun 2026), the setting is a controlled state-tracking problem with known transition rule

$N$04

where $N$05, $N$06, and $N$07. There, a model fine-tuned to emit intermediate states writes each $N$08 explicitly in a visible scratchpad. The intervention protocol edits the internal residual-stream representation of the printed current phase token at a chosen layer while leaving the visible scratchpad text fixed. The resulting “patch” tests whether later computation follows the edited latent state rather than the original visible text. In Qwen2.5-Coder-7B, the running-state model achieves edited-branch agreement $N$09 on the $N$10 task and $N$11 on the $N$12 task, while pretrained and final-answer-only controls remain near $N$13, and random or orthogonal patches yield $N$14 (Shih et al., 28 Jun 2026).

In "Notes-to-Self: Scratchpad Augmented VLAs for Memory Dependent Manipulation Tasks" (Haresh et al., 24 Feb 2026), scratchpad patching refers to updating an external sequence of text tokens $N$15 that is prepended or interleaved with the current image and language goal at each timestep. The policy outputs an action $N$16 and a language description $N$17, and the scratchpad update is governed by a gating bit $N$18: if $N$19 includes an update trigger such as $N$20, then $N$21; otherwise the scratchpad is left unchanged. In the reported ClevrSkills-Mem experiments, average success across five tasks rises from approximately $N$22 to approximately $N$23 for T-VLA with SP, and from approximately $N$24 to approximately $N$25 for R-VLA with SP (Haresh et al., 24 Feb 2026).

These usages are not identical to the byte-level SP mechanism. In the byte-level paper, scratchpads are transient internal summaries inside a patch; in the causal-reasoning paper, patching is an activation intervention on a written intermediate state; in the VLA paper, patching is an explicit update rule for an external text memory. What unifies them is the attempt to make intermediate state representations operational rather than merely decorative. This suggests that “scratchpad” functions as a broader design motif across multiple subfields, while “Scratchpad Patching (SP)” in the strict acronymized sense refers specifically to the byte-level patch-based mechanism of (Zheng et al., 10 May 2026).

6. Significance, limitations, and interpretive scope

The principal technical significance of SP in byte-level language modeling is that it reframes the compute-quality trade-off of patch-based models. Larger committed patches no longer imply a single fixed amount of within-patch computation, because scratchpads permit selective refresh at high-entropy positions. The paper therefore describes SP as decoupling compute from patch size, rather than replacing patching itself (Zheng et al., 10 May 2026).

The ablations also identify clear limits. Entropy-based triggers outperform fixed-stride and whitespace heuristics in BPB versus FLOPs for a Fixed $N$26 base. Dense updates are not uniformly beneficial. Under H-Net, scratchpad triggers often land one byte before H-Net boundaries, causing redundant near-boundary computations, which explains smaller gains under strict FLOPs matching. These findings indicate that SP is not a universal monotonic improvement; its behavior depends on interaction with the chosen patchifier and on threshold calibration (Zheng et al., 10 May 2026).

The related reasoning and robotics papers sharpen a broader conceptual point. In the reasoning study, “causal legibility” is distinguished from “plausible legibility”: it is not enough for a model to write a plausible chain of thought if later computation does not use the written variables (Shih et al., 28 Jun 2026). In the VLA setting, the scratchpad serves as a persistent textual memory for object positions, plans, and progress toward subgoals, and improvements are largest on memory-dependent long-horizon tasks (Haresh et al., 24 Feb 2026). A plausible implication is that contemporary scratchpad research is converging on a shared criterion: intermediate states become scientifically useful when they alter downstream computation in predictable ways.

Within that broader landscape, Scratchpad Patching in the sense of (Zheng et al., 10 May 2026) occupies a specific systems and architecture niche. It is a method for improving byte-level autoregressive modeling under sequence compression by repairing stale intra-patch context, using transient, entropy-triggered summaries that are computationally active but not persistently cached.