Chonkers Algorithm: Deterministic CDC

• Chonkers Algorithm is a deterministic content-defined chunking method that processes pre-segmented proto-chunks to enforce strict size and edit-locality guarantees.
• It uses a hierarchical, multi-phase approach—balancing, caterpillar, and diffbit phases—to systematically merge chunks and constrain edit propagation.
• The algorithm underpins applications in deduplication, versioning, and canonical string representations (Yarn) with rigorous worst-case formal guarantees.

Searching arXiv for the primary paper and closely related CDC references mentioned in the provided data. arXiv search query: "ti:Chonkers Algorithm" Chonkers is a deterministic content-defined chunking (CDC) algorithm designed to provide, simultaneously, strict chunk-size guarantees and strict edit-locality guarantees. Rather than selecting boundaries by rolling hashes or anchor predicates over a moving window, it starts from an input already divided into proto-chunks and repeatedly merges adjacent chunks through a hierarchical layered procedure. The algorithm’s defining mechanisms are a balancing phase for locally light chunks, a caterpillar phase for exact adjacent repetition, and a diffbit phase that assigns deterministic local merge priorities from recursively defined first-difference descriptors. Its stated application domain includes deduplication, versioning, substring matching, and canonical persistent string representations such as Yarn (Berger, 14 Sep 2025).

1. Position within content-defined chunking

In CDC, boundaries are determined by content rather than fixed offsets, so insertions and deletions need not globally destroy alignment. Chonkers is motivated by a tradeoff that the paper identifies in prior CDC families. Rabin-based CDC is described as having what the paper calls strict locality, because unchanged content far from an edit keeps its boundaries unchanged, but its chunk sizes are only guaranteed in expectation and pathological inputs can yield very small or very large chunks. Anchor-based chunkers such as Gear and FastCDC are described as fast and usually reasonable in chunk-size distribution, but without a formal bounded-propagation guarantee for insertions or deletions (Berger, 14 Sep 2025).

This places Chonkers at a different point in the CDC design space from recent throughput-oriented methods. SeqCDC uses monotonic byte sequences and content-defined skipping to increase chunking throughput, especially for large chunk sizes (Udayashankar et al., 27 May 2025). VectorCDC accelerates hashless extrema-based CDC using SSE and AVX while preserving exact chunk boundaries (Udayashankar et al., 7 Aug 2025). These approaches emphasize scan-rate and SIMD efficiency; Chonkers instead targets deterministic worst-case structure and locality.

2. Layered construction and chunk classes

Chonkers operates through layers indexed by $i$. If $A_i$ is the layer’s absolute unit, then

$A_i < A_{i+1} \le 2A_i.$

The paper states that exact doubling is optimal for the strongest guarantees, while larger jumps weaken size guarantees and can weaken locality bounds (Berger, 14 Sep 2025).

Relative to the current layer’s absolute unit, chunks are classified by normalized weight. The paper defines a chunk’s weight as the number of bits it contains, denoted informally as “$\#1\,c$ bits.”

Class	Weight range
Megachonkers	$\ge 1$
Heftychonks	$\ge \frac12$ and $<1$
Fine bois	$\ge \frac14$ and $<\frac12$
Kittens	$< \frac14$

Two adjacent chunks $A_i$0 are heckin’ if they can be merged without creating a megachonker: $A_i$1 The boundary between them is then heck’d. The algorithm also introduces caterpillars, which are runs of two or more equal adjacent chunks. A caterpillar may exceed weight $A_i$2 provided its repeated unit, called the segment, is lighter than $A_i$3. This exception is structural rather than accidental: periodicity is treated as an explicit object in the chunk representation.

Each layer executes three phases in order: balancing, caterpillar, and diffbit. The output of one phase becomes the input of the next, and the postconditions of each phase serve as preconditions for the next phase.

3. Balancing, caterpillars, and diffbits

The balancing phase is intended to eliminate problematic very small chunks before more content-sensitive merging begins. A chunk lighter than all its neighbors is considered for merging with its right neighbor at priority $A_i$4 and with its left neighbor at priority $A_i$5. The formal precondition is that there are no two consecutive kittens. The stated postcondition is that no kitten is also a heckin’ chonker: after balancing, any surviving kitten cannot be merged with either neighbor without exceeding the unit bound (Berger, 14 Sep 2025).

The caterpillar phase then scans for consecutive chunks with exactly equal bit content and merges as many as possible into a caterpillar. This preserves the balancing postcondition and ensures that there are no two consecutive equal chunks that are also heckin’ with each other. The reason is technical as well as representational: the later diffbit construction is only defined between unequal bit sequences.

The diffbit phase is the algorithm’s most distinctive component. For two unequal bit sequences, the diffbit is defined as

$A_i$6

where $A_i$7 is the zero-based index of the first differing bit, and $A_i$8 records the direction of change: $A_i$9 if the left sequence has $A_i < A_{i+1} \le 2A_i.$0 and the right sequence has $A_i < A_{i+1} \le 2A_i.$1, and $A_i < A_{i+1} \le 2A_i.$2 in the opposite case. Chonkers computes this on augmented bit content, formed by prepending a binary encoding of the chunk’s weight and, optionally, a hash code to the raw bit content. For adjacent chunks $A_i < A_{i+1} \le 2A_i.$3 that are heckin’, the diffbit of their augmented contents is assigned to the left chunk as its first-order diffbit.

If a chunk is not heckin’ with its right neighbor, Chonkers still assigns a diffbit using a fictitious comparison that differs at bit $A_i < A_{i+1} \le 2A_i.$4. This is explicitly introduced to reduce causal spread: non-heck’d boundaries should not allow right-side content to influence left-side decisions. The paper proves that any two consecutive diffbits are distinct, which is what permits recursive refinement.

Higher-order diffbits are defined recursively. If $A_i < A_{i+1} \le 2A_i.$5 is the $A_i < A_{i+1} \le 2A_i.$6-th order diffbit of chunk $A_i < A_{i+1} \le 2A_i.$7, then

$A_i < A_{i+1} \le 2A_i.$8

where $A_i < A_{i+1} \le 2A_i.$9 is the right neighbor or the fictitious substitute. The highest-order diffbit used becomes the merge priority of the boundary between $\#1\,c$0 and its right neighbor, provided that boundary is heck’d. The number of required diffbit orders is controlled by the recurrence

$\#1\,c$1

which yields growth sufficient for the paper’s conclusion that the number of required orders is

$\#1\,c$2

The paper states that in realistic applications this is at most about $\#1\,c$3–$\#1\,c$4 (Berger, 14 Sep 2025).

4. Deterministic merge system and formal guarantees

Both the balancing phase and the diffbit phase use the same deterministic merge mechanism. A boundary with priority $\#1\,c$5 is processed at that priority if two rules permit it. The first is the no-megachonkers rule: $\#1\,c$6 Only heck’d boundaries may be removed. The second is the determinism rule: if the right boundary of the right chunk also has priority $\#1\,c$7, the current merge is disallowed. Equivalently, in a consecutive run of equal-priority candidate merges, only the rightmost one may be performed. When two chunks merge, the resulting chunk inherits the left boundary priority of the left constituent and the right boundary priority of the right constituent, even if those inherited boundaries are no longer heck’d.

The size guarantees are layered and inductive. After balancing, no surviving kitten is mergeable with either neighbor. Caterpillar formation preserves this property. Given the caterpillar-phase preconditions, the diffbit phase guarantees that no two consecutive chunks have weight $\#1\,c$8. In the paper’s terminology, no two consecutive fine bois or kittens remain adjacent. Under the layer condition $\#1\,c$9, these per-layer guarantees compose.

The paper’s final size statements are explicit (Berger, 14 Sep 2025):

1. Megachonkers can occur only if they were overweight proto-chunks or if they are caterpillars with segment weight $\ge 1$0.
2. For any two consecutive chunks, at most one may be lighter than $\ge 1$1.
3. If a chunk has weight $\ge 1$2, then with either neighbor its pair weight is at least $\ge 1$3.
4. Excluding one end chunk if necessary, the worst-case average weight is lower-bounded by alternating $\ge 1$4 and $\ge 1$5 chunks, giving

$\ge 1$6

The locality guarantees are likewise deterministic. The paper analyzes four influence sources: balancing-phase priority dependence, diffbit computation, merge cascades caused by the no-megachonkers and determinism rules, and compounding across layers. Within one layer, the paper gives influence bounds of up to 9 absolute units to the right and 12 absolute units to the left. Under doubling layers, lower-layer effects shrink geometrically in final-layer units, so the total worst-case bounds become

$\ge 1$7

The intended significance is that a single edit perturbs chunking only in a bounded neighborhood, and that this is a deterministic worst-case statement rather than an average-case or probabilistic one.

5. Data structures, streaming realization, and Yarn

The implementation described in the paper maintains a DAG of chunk derivations. Ordinary merged chunks point to two children, while caterpillars point to repeated constituents or a compressed run-length representation. Chunks are deduplicated as soon as they are formed, typically using a hash table. Each chunk stores or can derive its weight, content hash, phase and layer of formation, child references, and relevant boundary priorities.

Efficient diffbit computation is a central engineering issue, because naive first-difference search over recursively merged content would be expensive. The paper proposes combining structural deduplication, prepended hash codes in the augmented content, and recursive descent only into differing regions. It also suggests concatenation-homomorphic hashes, including SL$\ge 1$8-based hashes or Rabin fingerprints, so that parent hashes can be computed from child hashes and lengths without rescanning content (Berger, 14 Sep 2025).

The algorithm is not described as inherently offline. The paper sketches a streaming architecture in which diffbit-priority computation buffers up to seven chunks, each priority-based merge processor buffers two chunks, balancing priority assignment buffers three chunks, and caterpillar formation buffers two chunks. It also states that bounded influence allows parallel chunking on overlapping segments, with special handling for caterpillars that cross segment boundaries.

Chonkers is also the basis of the Yarn datatype. Yarn is a string representation built on a chonker tree whose leaves are characters. It is described as similar in spirit to a rope, but with a decisive difference: the tree structure is unique and content-determined, not dependent on construction history. Because Chonkers is deterministic and edit-local, the same string yields the same canonical tree shape. The paper states that basic Yarn operations run in expected

$\ge 1$9

time, and that equality can be trivial because identical content implies identical canonical roots (Berger, 14 Sep 2025).

6. Empirical behavior, comparisons, and limitations

The experiments reported for Chonkers use three corpora: random strings, Linux kernel C/H files, and German fiction texts. Empirically, average chunk weight is described as very stable around

$\ge \frac12$0

absolute units across many layers and corpora, which is substantially above the theoretical lower bound of $\ge \frac12$1. The maximum segment weight never exceeded $\ge \frac12$2, even when average chunk weight exceeded $\ge \frac12$3 in periodic-heavy data such as kernel-source indentation, supporting the claim that overweight chunks in practice arise as caterpillars with bounded segment size. The minimum adjacent pair weight remained at or above approximately $\ge \frac12$4, consistent with the structural guarantees (Berger, 14 Sep 2025).

For single-character deletions, empirical locality was substantially tighter than the proof constants. The paper reports that average propagation was well below $\ge \frac12$5 absolute unit and that observed maxima did not exceed $\ge \frac12$6 absolute units in the reported experiments. A phase census indicates that most merges occur in balancing phase, priority $\ge \frac12$7 and diffbit phase, priority $\ge \frac12$8, with balancing priority $\ge \frac12$9 becoming more common in later layers.

In comparison with other CDC families, Chonkers is presented as preferable when worst-case guarantees matter, especially for canonical persistent string representations, incremental recomputation under local edits, deduplicated versioned storage, and substring-oriented structures. By contrast, recent chunkers such as SeqCDC and acceleration frameworks such as VectorCDC are oriented toward raw chunking throughput, SIMD efficiency, and large-scale backup or deduplication pipelines (Udayashankar et al., 27 May 2025, Udayashankar et al., 7 Aug 2025). A plausible implication is that these approaches are complementary rather than mutually exclusive: the Chonkers paper explicitly suggests using a fast probabilistic CDC as a first stage and invoking Chonkers to repair problematic oversized or irregular regions (Berger, 14 Sep 2025).

The paper also identifies several limitations. The reference implementation is stated to be not optimized for maximum performance. Compared with Rabin- or anchor-based CDC, Chonkers is algorithmically more involved because it requires three phases per layer, recursive diffbits, caterpillar handling, priority inheritance, DAG deduplication, and incremental rebuild logic. The locality constants $<1$0 right and $<1$1 left are acknowledged as conservative, and the strongest locality proof depends on approximate doubling of layer units. Efficient caterpillar storage and compact serialization are left as open engineering problems. The result is a CDC method whose distinctive contribution is not raw scan speed, but the replacement of anchor scanning by a deterministic, layered merge system that simultaneously constrains chunk size and edit propagation (Berger, 14 Sep 2025).