Midicoth: Micro-Diffusion in Lossless Compression

• Midicoth is a lossless compression system featuring a micro-diffusion denoising layer that corrects biases from Jeffreys smoothing in adaptive statistical models.
• It decomposes a 256-way byte prediction into a hierarchy of binary calibration tasks, thereby replacing one complex calibration problem with sequential binary ones to improve data efficiency.
• Experimental results show notable compression improvements over xz -9 on various datasets, demonstrating its capability for effective bias correction and online adaptability.

Midicoth is a lossless compression system that introduces a micro-diffusion denoising layer for improving probability estimates produced by adaptive statistical models. Its central claim is that, in compressors such as Prediction by Partial Matching (PPM), prior smoothing handles sparse observations but also produces distributions that are significantly flatter than the true source distribution when contexts have been seen only a few times; Midicoth treats this as a shrinkage process and applies a reverse denoising step that corrects predicted probabilities using empirical calibration statistics. To make this correction data-efficient, it decomposes each byte prediction into a hierarchy of binary decisions along a bitwise tree, replacing a single 256-way calibration problem with a sequence of binary calibration tasks. The denoising layer is applied as a lightweight post-blend calibration stage after all model predictions have been combined, and the complete system combines five fully online components (Tacconelli, 9 Mar 2026).

1. System architecture

For each byte position $i$, Midicoth produces a 256-way distribution $P(s)$ that is passed to an arithmetic coder. The pipeline has five fully online components in a fixed cascade, and the micro-diffusion layer is always applied last, after all blending, to correct the entire ensemble’s residual biases (Tacconelli, 9 Mar 2026).

Component	Role
Adaptive PPM	Orders 0–4 with PPMC-style exclusion and Jeffreys prior
Long-range match model	Multiple context lengths with continuation tracking
Trie-based word model	Word continuation and bigram for first-byte-of-next-word
High-order context model	Orders 5–8, separate from PPM, minimal smoothing
Micro-diffusion layer	Binary-tree Tweedie denoising, $K=3$ successive steps

The coding backend is a 32-bit arithmetic coder (Witten–Neal–Cleary) with a 14-bit frequency scale $T=16384$. Probabilities are quantized as

$f(s) = \max(1, \lfloor P(s)\cdot T + 0.5 \rfloor),$

and the cumulative frequency table is then fed to the coder. Encoder and decoder use the exact same online models, which guarantees determinism and bit-exact reconstruction.

This design places Midicoth within the tradition of adaptive, byte-level statistical compression, but with the distinctive addition of a final calibration layer that operates on the fully blended distribution rather than on any single submodel. A plausible implication is that the method targets systematic bias that survives model combination rather than merely improving one component in isolation.

2. Shrinkage, Jeffreys smoothing, and the denoising premise

Midicoth formalizes prior smoothing as shrinkage. For a multinomial with counts $n_i$ over 256 symbols, the Jeffreys prior is a symmetric Dirichlet with $0.5$ per symbol, giving total $\alpha=128$. The smoothed estimate is

$$p_i = \frac{n_i + \alpha q_i}{N + \alpha}, \qquad q_i = \frac{1}{256}, \qquad \alpha = 128,$$

so concretely

$p_i = \frac{n_i + 0.5}{N + 128},$

where $P(s)$0 (Tacconelli, 9 Mar 2026).

In expectation, if the true distribution is $P(s)$1, then

$P(s)$2

with $P(s)$3 and

$P(s)$4

When $P(s)$5 is small, $P(s)$6 is small and predictions are pulled toward uniform. The effective “noise fraction” of this shrinkage is

$P(s)$7

This $P(s)$8 is the quantity Midicoth uses to condition the denoiser, as an analog of $P(s)$9 in Tweedie or diffusion formulations.

The underlying PPM component uses Prediction by Partial Matching with Method C escapes and exclusion:

$K=3$0

with

$K=3$1

Here $K=3$2 is the set of symbols assigned at higher orders, and $K=3$3 is the real (post-prior) count mass available at order $K=3$4 for non-excluded symbols. Exclusion stops wasting mass on symbols already explained up the chain.

The conceptual move made by Midicoth is to interpret the flattening caused by Jeffreys smoothing as a forward noising process. This suggests a reverse step that estimates how much probability mass should be restored to sharper, empirically supported alternatives.

3. Binary-tree Tweedie denoising

The micro-diffusion layer views Jeffreys smoothing as a convex mixture

$K=3$5

with the “noise level” $K=3$6 determined online by the matched context’s total count $K=3$7. Midicoth then adopts Tweedie empirical Bayes as an operational analogy. For additive Gaussian noise $K=3$8, $K=3$9, Tweedie’s formula gives

$T=16384$0

In Midicoth, the forward noise is not Gaussian but shrinkage toward uniform, so the score-based correction is implemented empirically rather than analytically (Tacconelli, 9 Mar 2026).

At each binary node, Midicoth maintains a discretized calibration bin $T=16384$1 with

• $T=16384$2,
• $T=16384$3,
• $T=16384$4,
• $T=16384$5,

where $T=16384$6 is the observed left/right decision. The additive Tweedie correction is

$T=16384$7

This approximates $T=16384$8.

To avoid overfitting in low-count bins, Midicoth uses James–Stein shrinkage for robustness:

$T=16384$9

$f(s) = \max(1, \lfloor P(s)\cdot T + 0.5 \rfloor),$0

$f(s) = \max(1, \lfloor P(s)\cdot T + 0.5 \rfloor),$1

and if $f(s) = \max(1, \lfloor P(s)\cdot T + 0.5 \rfloor),$2, it sets $f(s) = \max(1, \lfloor P(s)\cdot T + 0.5 \rfloor),$3. This variance-aware attenuation suppresses noisy corrections early and becomes permissive once bins have high SNR.

A byte $f(s) = \max(1, \lfloor P(s)\cdot T + 0.5 \rfloor),$4 with bits $f(s) = \max(1, \lfloor P(s)\cdot T + 0.5 \rfloor),$5 is represented as a path of binary decisions $f(s) = \max(1, \lfloor P(s)\cdot T + 0.5 \rfloor),$6 from MSB to LSB:

$f(s) = \max(1, \lfloor P(s)\cdot T + 0.5 \rfloor),$7

The 256-way probability factors as

$f(s) = \max(1, \lfloor P(s)\cdot T + 0.5 \rfloor),$8

Given the current 256-way distribution $f(s) = \max(1, \lfloor P(s)\cdot T + 0.5 \rfloor),$9, the probability of going right at an internal node is

$n_i$0

At each node, the corrected probability is

$n_i$1

This is converted into scale factors

$n_i$2

which are propagated multiplicatively down the tree, followed by renormalization of the 256 leaf probabilities.

The paper characterizes this as “binary-tree Tweedie denoising.” Its data-efficiency comes from replacing a single 256-class calibration problem with eight binary ones, while preserving consistency of the final 256-way distribution through subtree scaling and renormalization.

4. Successive refinement and online calibration

Midicoth uses $n_i$3 independent denoising steps. If $n_i$4 is the blended pre-denoise distribution, then for $n_i$5 the system builds the sum tree over $n_i$6, looks up $n_i$7 from calibration tables, shrinks it to $n_i$8 via the SNR rule, applies tree scales, and renormalizes to obtain $n_i$9 (Tacconelli, 9 Mar 2026). In compact residual form,

$0.5$0

Empirically, $0.5$1 gives the largest correction, while $0.5$2 provide smaller residual gains. The paper states that $0.5$3 adds negligible benefit but increases table sparsity and adaptation latency.

Each calibration entry is indexed by

• step $0.5$4,
• bit context $0.5$5,
• order group $0.5$6 grouping PPM orders $0.5$7, $0.5$8, and $0.5$9,
• shape $\alpha=128$0 discretized by current $\alpha=128$1 thresholds $\alpha=128$2,
• confidence $\alpha=128$3, log-spaced bins of matched PPM context count $\alpha=128$4,
• probability bin $\alpha=128$5, logit-spaced in $\alpha=128$6.

This yields

$\alpha=128$7

entries, requiring approximately $\alpha=128$8 MB because each entry stores four doubles. Each entry starts with $\alpha=128$9 pseudo-observations at the bin center probability $$p_i = \frac{n_i + \alpha q_i}{N + \alpha}, \qquad q_i = \frac{1}{256}, \qquad \alpha = 128,$$0:

$$p_i = \frac{n_i + \alpha q_i}{N + \alpha}, \qquad q_i = \frac{1}{256}, \qquad \alpha = 128,$$1

These controls make the early stream deliberately conservative. This suggests that Midicoth’s calibration layer is not a static post-training module but an adaptive online estimator whose own uncertainty is explicitly managed through pseudocounts, SNR shrinkage, and low-count cutoffs.

5. Backend models, blending, and deterministic operation

The denoiser is only the final stage of a broader ensemble. The long-range match model predicts a symbol $$p_i = \frac{n_i + \alpha q_i}{N + \alpha}, \qquad q_i = \frac{1}{256}, \qquad \alpha = 128,$$2 with

$$p_i = \frac{n_i + \alpha q_i}{N + \alpha}, \qquad q_i = \frac{1}{256}, \qquad \alpha = 128,$$3

and blends it as

$$p_i = \frac{n_i + \alpha q_i}{N + \alpha}, \qquad q_i = \frac{1}{256}, \qquad \alpha = 128,$$4

with $$p_i = \frac{n_i + \alpha q_i}{N + \alpha}, \qquad q_i = \frac{1}{256}, \qquad \alpha = 128,$$5, where $$p_i = \frac{n_i + \alpha q_i}{N + \alpha}, \qquad q_i = \frac{1}{256}, \qquad \alpha = 128,$$6 is derived from length and continuation streak. The trie-based word model supplies word continuation during a word and a bigram for the first byte of the next word; its blend weight satisfies $$p_i = \frac{n_i + \alpha q_i}{N + \alpha}, \qquad q_i = \frac{1}{256}, \qquad \alpha = 128,$$7. The high-order context model uses orders $$p_i = \frac{n_i + \alpha q_i}{N + \alpha}, \qquad q_i = \frac{1}{256}, \qquad \alpha = 128,$$8–$$p_i = \frac{n_i + \alpha q_i}{N + \alpha}, \qquad q_i = \frac{1}{256}, \qquad \alpha = 128,$$9 with

$p_i = \frac{n_i + 0.5}{N + 128},$0

confidence

$p_i = \frac{n_i + 0.5}{N + 128},$1

and blend weight

$p_i = \frac{n_i + 0.5}{N + 128},$2

(Tacconelli, 9 Mar 2026).

Encoder and decoder share the same high-level per-byte pipeline: initialize all models; obtain a 256-way distribution from PPM together with total count $p_i = \frac{n_i + 0.5}{N + 128},$3 and matched order $p_i = \frac{n_i + 0.5}{N + 128},$4; normalize; blend match, word, and high-order predictions; denoise with micro-diffusion; quantize to cumulative frequencies; encode or decode with the arithmetic coder; and then update all models with the observed byte. Because the same online state transitions occur on both sides, the process remains symmetric and deterministic.

The implementation reported in the paper is a single-core C implementation at approximately $p_i = \frac{n_i + 0.5}{N + 128},$5 KB/s, with roughly $p_i = \frac{n_i + 0.5}{N + 128},$6 lines of code and calibration tables of approximately $p_i = \frac{n_i + 0.5}{N + 128},$7 MB. The open-source code is available at https://github.com/robtacconelli/midicoth.

6. Results, ablations, and relation to prior compressors

Midicoth reports the following main results (Tacconelli, 9 Mar 2026). On alice29.txt (152 KB), it achieves $p_i = \frac{n_i + 0.5}{N + 128},$8 bpb (26.5%), beating xz -9 at $p_i = \frac{n_i + 0.5}{N + 128},$9 bpb by $P(s)$00. On enwik8 (100 MB), it achieves $P(s)$01 bpb (21.9%), beating xz -9 at $P(s)$02 bpb by $P(s)$03. On English Indices of Deprivation 2025 (334 KB, out-of-distribution), it achieves $P(s)$04 bpb, beating xz -9 at $P(s)$05 bpb by $P(s)$06.

Component ablations attribute marginal gains as follows: the match model contributes $P(s)$07 on alice29, $P(s)$08 on enwik8_3M, and $P(s)$09 on EID 2025; the word model contributes $P(s)$10–$P(s)$11; the high-order context model contributes $P(s)$12–$P(s)$13; and the micro-diffusion post-blend stage contributes $P(s)$14–$P(s)$15 consistently.

The paper positions Midicoth against two main baselines. Relative to standard PPM smoothing, it explicitly reverses shrinkage toward uniform, conditioned on $P(s)$16, and does so after blending, thereby capturing biases introduced not only by PPM but also by the match, word, and high-order models. Relative to PAQ/CMIX-style post-blend calibration, it is presented as a lightweight alternative that performs nonparametric score estimation and additive correction at binary nodes rather than using neural mixers and bitwise SSE. The paper further reports that chaining SSE after Tweedie hurts because they compete for the same calibration signal, and that Tweedie alone is better in this setting.

The paper also states where the method helps less. On binary or non-text data, word and high-order models contribute less, and micro-diffusion still calibrates but overall gain is smaller. Very tiny files, specifically below $P(s)$17 KB, provide too few observations to populate calibration tables, so James–Stein shrinkage suppresses most corrections. Recommended defaults are Jeffreys prior with $P(s)$18 per symbol, $P(s)$19 denoising steps, $P(s)$20 bit contexts, $P(s)$21 order groups, $P(s)$22 shape bins, $P(s)$23 confidence bins, $P(s)$24 probability bins, $P(s)$25 pseudocounts per entry, SNR threshold $P(s)$26, low-count cutoff $P(s)$27, clamp $P(s)$28, and arithmetic-coder scale $P(s)$29.

The novelty claimed for Midicoth is therefore specific: it treats online Jeffreys smoothing as a forward noise process and implements a score-based reverse step with Tweedie-inspired additive corrections at the nodes of a binary tree. A plausible implication is that its main contribution is not a new base model class, but a calibration mechanism that exploits the structure of byte prediction to make online empirical Bayes correction feasible at compressor time scales.