Byte-Exact Chunk-Level Deduplication

• Byte-exact chunk-level deduplication is a storage technique that partitions data into fixed or content-defined chunks and uses strong fingerprints like SHA-1 to retain only one physical copy of each exact duplicate.
• It addresses challenges in chunk formation, boundary selection, and metadata consistency in distributed systems to enhance backup performance and system reliability.
• Recent advances integrate accelerated chunking, encrypted deduplication, and inference-time optimization to boost throughput while balancing security and deduplication efficiency.

Searching arXiv for recent and foundational papers on byte-exact chunk-level deduplication, content-defined chunking, encrypted deduplication, and distributed deduplication. Byte-exact chunk-level deduplication is a storage and data-management technique in which an input byte stream is partitioned into chunks, each chunk is identified by a content-derived fingerprint, and only one physical copy of each exactly matching chunk is retained. In this model, deduplication is strictly lossless and exact at chunk granularity: two chunks are deduplicated only when they are treated as identical byte sequences, typically via equality of a strong content fingerprint such as SHA-1 or a cryptographic chunk hash (Khan et al., 2018). Across the literature, the term encompasses both fixed-size chunking and content-defined chunking (CDC), but the central invariant remains unchanged: chunk identity is content-based, not name-based or location-based, and duplicate elimination operates at the level of exact chunk equality rather than approximate similarity (Niesen, 2017). The topic sits at the intersection of distributed storage architecture, chunking algorithms, encryption, information theory, and, more recently, inference-time context optimization for LLMs (Khan et al., 2018, Gregoriadis et al., 2024, Schelpe, 11 May 2026).

1. Definition and conceptual scope

Byte-exact chunk-level deduplication treats a data object as a sequence of chunks and replaces repeated chunks by references to a single stored instance. In the cluster-wide Ceph design of Liu et al., the deduplication unit is a fixed-size data chunk, and the receiving storage server is “responsible for splitting the object into small fixed-size data chunks” (Khan et al., 2018). Each chunk is then assigned a content fingerprint computed from the chunk contents, specifically

$h(c)=\text{SHA-1}(c),$

and the fingerprint functions simultaneously as the deduplication key and as the placement key for distributed storage (Khan et al., 2018).

The literature also formalizes byte-exact chunk deduplication as a special case of broader source-coding frameworks. In generalized deduplication, classic deduplication arises when each chunk is its own base and the deviation set contains only the all-zero chunk, so that there is no similarity model beyond exact equality; formally, $\mathcal{Y}=\{\boldsymbol{0}\}$, $\phi(z)=z$, and $q=0$ (Vestergaard et al., 2019). This establishes exact chunk matching as the degenerate but operationally important case in which all deduplication benefit derives from repeated identical chunks rather than near-duplicate structure (Vestergaard et al., 2019).

An information-theoretic treatment likewise models byte-exact deduplication as dictionary-based coding over equal binary substrings. In that framework, the observed source sequence $\mathsf{S}$ is parsed into chunks $Z_c$, and the encoder either emits a new chunk verbatim or emits a pointer to a previous identical chunk (Niesen, 2017). This formulation makes the byte-exactness assumption precise: the scheme exploits only equality of chunk strings and does not rely on semantic equivalence, approximate matching, or delta coding (Niesen, 2017).

A more recent line of work transfers the same concept to record- or passage-level text pipelines. Merlin defines byte-exact equivalence for finite byte sequences $c_i$ and $c_j$ by equality of length and equality of every byte position, and then retains only the first occurrence of each equivalence class in order (Schelpe, 11 May 2026). That work operates at application-defined record granularity rather than file-system chunks, but the underlying semantics remain identical: duplicate elimination is deterministic and byte-for-byte exact (Schelpe, 11 May 2026).

2. Chunk formation and boundary selection

The principal design distinction in byte-exact chunk-level deduplication is not whether matching is exact, but how chunk boundaries are chosen. The oldest and simplest option is fixed-size chunking, in which boundaries are placed at constant offsets. This yields deterministic byte-exact chunk identities, but it is brittle under insertions and deletions because a one-byte shift can misalign all subsequent chunk boundaries (Udayashankar et al., 27 May 2025, Udayashankar et al., 7 Aug 2025).

Content-defined chunking addresses this alignment problem by deriving boundaries from the byte stream itself. In the standard hash-based formulation, a sliding window is scanned and a boundary is declared when a rolling hash satisfies a mask predicate such as

$H_i \,{data}\, (2^b-1)=0,$

or more generally when $(H_i {data} M)=V$ (Udayashankar et al., 7 Aug 2025, Gregoriadis et al., 2024). Rabin, Buzhash, and Gear instantiate this family with different rolling-hash constructions, but all preserve the same byte-exact semantics: chunk boundaries are deterministic functions of the content, and deduplication itself is subsequently enforced by chunk fingerprints rather than by the boundary detector (Gregoriadis et al., 2024, Udayashankar et al., 7 Aug 2025).

The contemporary CDC literature has also developed hashless boundary detectors. SeqCDC is a sequence-based CDC algorithm that declares a boundary at the end of a monotonic sequence of bytes of length SeqLength, subject to minimum and maximum chunk-size constraints (Udayashankar et al., 27 May 2025). In increasing mode, the boundary condition is

$\mathcal{Y}=\{\boldsymbol{0}\}$0

with a boundary placed at position $\mathcal{Y}=\{\boldsymbol{0}\}$1 (Udayashankar et al., 27 May 2025). VectorCDC accelerates related hashless CDC schemes such as AE, RAM, and MAXP via SIMD while preserving identical boundary positions relative to their scalar definitions (Udayashankar et al., 7 Aug 2025). These works are significant because they separate two questions that are often conflated: byte-exact deduplication requires deterministic chunk identities, but it does not require rolling hashes in the boundary detector (Udayashankar et al., 27 May 2025, Udayashankar et al., 7 Aug 2025).

Theoretical work has clarified why boundary choice matters so strongly. In the information-theoretic analysis of deduplication, fixed-length chunking can become catastrophically suboptimal when underlying repeated source blocks have slightly varying lengths, because chunk boundaries lose synchronization with the actual repeated structures (Niesen, 2017). Variable-length, anchor-based chunking re-establishes synchronization and can avoid the linear-factor inefficiency that afflicts fixed-size chunking under boundary drift (Niesen, 2017). This suggests that byte-exactness alone is insufficient as a design criterion: exact chunk identity is necessary, but deduplication efficiency depends critically on whether the chunking method preserves alignment of unchanged data across edits.

The recent comparative study of CDC algorithms reaches a closely related empirical conclusion. It finds that FSC remains vulnerable to the boundary-shift problem, whereas CDC algorithms maintain higher deduplication ratios on workloads with insertions, deletions, and version churn (Gregoriadis et al., 2024). It further shows that algorithmic families differ not only in throughput but in mean chunk size and chunk-size variance, both of which affect the storage overhead and the granularity of exact matching (Gregoriadis et al., 2024).

3. Fingerprints, metadata, and exact identity

In byte-exact chunk-level deduplication, chunk identity is represented through metadata structures that map a content fingerprint to stored data and to reconstruction information. In the Ceph-based shared-nothing design, each OSD embeds a Deduplication Metadata Shard (DM-Shard) containing two principal structures: OMAP, which stores object reconstruction information, and the Chunk Information Table (CIT), which stores per-chunk deduplication metadata (Khan et al., 2018). OMAP records the object name, object fingerprint, and the list of chunk fingerprints composing the object, while CIT maintains the chunk fingerprint, a reference count, and a commit flag indicating whether the chunk is valid (Khan et al., 2018).

The functional separation between reconstruction metadata and deduplication metadata is conceptually important. OMAP is an object-to-chunk mapping; it is not itself the deduplication index. CIT is the core exact-match index, consulted on each chunk write to determine whether a fingerprint already exists, whether the chunk is valid, and whether the reference count should be incremented or a new physical chunk should be written (Khan et al., 2018). The implementation uses SQLite as the DM-Shard backend, making the shard effectively a local key-value or relational index (Khan et al., 2018).

The same logical architecture recurs in other systems. RevDedup maintains chunk and segment fingerprints, offsets, and backup recipes, with SHA-1 used as the identifying fingerprint for both segments and chunks (Li et al., 2014). Its coarse inline segment-level deduplication and fine reverse chunk-level deduplication differ in timing and granularity, but both remain byte-exact because all matching is performed by cryptographic fingerprints over chunk contents, with exact duplicates only (Li et al., 2014). Content-defined Merkle Trees (CDMT) extend the metadata concept further by organizing chunk hashes into a version-comparison index over deduplicated storage; leaves are chunk hashes, and internal nodes are content-defined hashes over child hashes, enabling logarithmic-time change detection on the client (Nakamura et al., 2021).

In recent inference-time deduplication systems, the same principle appears in a simplified setting. Merlin deduplicates an ordered list of records or chunks using a deterministic high-entropy digest and an open-addressing flat hash set, preserving first occurrence order (Schelpe, 11 May 2026). There, the metadata structure is transient and per-request rather than persistent and storage-backed, but the mechanics of exact identity remain the same: a fast fingerprint drives lookup, and on collision a deterministic byte comparison is used to preserve byte-exact correctness (Schelpe, 11 May 2026). That explicit fallback is notable because older storage papers often assume cryptographic collision probability is negligible and do not describe secondary byte-by-byte verification (Khan et al., 2018).

A recurring misconception is that rolling hashes used in CDC also define chunk identity. The literature is explicit that CDC algorithms select boundaries, while chunk equality is normally determined by a separate content fingerprint or cryptographic hash over the final chunk bytes (Udayashankar et al., 7 Aug 2025, Gregoriadis et al., 2024). This division of labor is fundamental: a boundary detector decides where chunks are, while the fingerprinting layer decides which chunks are identical.

4. Distributed and storage-system realizations

Byte-exact chunk-level deduplication becomes substantially more complex in distributed, shared-nothing storage systems because exact matching must coexist with scalability, decentralized metadata, rebalancing, and failure recovery. The Ceph implementation of cluster-wide deduplication provides a representative design. Chunk placement and deduplication metadata are distributed across the cluster using content-based placement:

$\mathcal{Y}=\{\boldsymbol{0}\}$2

so that the fingerprint determines both the physical chunk location and the home of the corresponding CIT entry (Khan et al., 2018). This removes the need for a central deduplication index and avoids broadcast lookups, because any node can compute directly which OSD is responsible for a chunk fingerprint (Khan et al., 2018).

This content-based sharding is logically significant because it preserves the design constraints of shared-nothing systems. Metadata and data co-move under CRUSH rebalancing, and no explicit location table must be updated when OSD membership changes (Khan et al., 2018). The paper states that this placement “relieves us from i) complicated location management for each data chunk, ii) modifications in existing self-balancing mechanism, and iii) frequent deduplication metadata updates” (Khan et al., 2018). As a result, byte-exact chunk deduplication does not require abandoning the self-balancing and fault-isolation properties of Ceph.

Backup systems impose a different set of system-level constraints, centered on restore performance and garbage collection rather than distributed lookup scalability. RevDedup addresses these by combining coarse-grained inline deduplication at segment level with fine-grained reverse deduplication at chunk level (Li et al., 2014). The exact chunk-level layer remains byte-exact, using SHA-1 fingerprints and variable-size Rabin chunking, but the reference direction is inverted for older backups: older backups are rewritten to point forward to newer chunk copies (Li et al., 2014). This design shifts fragmentation toward cold backups and keeps the latest backup largely sequential to restore. It also simplifies deletion because expired backups can be reclaimed by deleting timestamped containers instead of performing global mark-and-sweep over fine-grained shared chunks (Li et al., 2014).

CDMT applies chunk-level byte-exact deduplication to container images and version transport. The underlying chunk store deduplicates CDC-defined blocks exactly, while the CDMT structure indexes deduplicated blocks for efficient push and pull across image versions (Nakamura et al., 2021). The paper emphasizes that block-level deduplication alone does not improve push/pull efficiency because the system must still determine which blocks are already present on the client. CDMT addresses this by providing a content-defined internal-node structure that is robust to chunk shifts, unlike a traditional Merkle tree over CDC chunks (Nakamura et al., 2021). This demonstrates a broader systems lesson: exact deduplication storage and efficient change detection are related but distinct problems, and a practical deployment often requires an additional index over exact chunk identities.

5. Consistency, fault tolerance, and encrypted deduplication

Exact chunk identity is only useful if the mapping from fingerprint to stored bytes remains consistent under crashes and concurrent updates. In the Ceph cluster-wide design, the main failure mode is a partially committed chunk write: a CIT entry may exist even though the physical chunk is missing. To handle this, each CIT entry contains a commit flag with Flag ∈ {0,1} (Khan et al., 2018). New chunks are inserted with Flag=0, and an asynchronous consistency manager later sets Flag=1 when the write completes (Khan et al., 2018). On a duplicate lookup, if the fingerprint exists but Flag=0, the system performs a stat-like existence check on the chunk; if the chunk is missing, it rewrites the data before allowing new references (Khan et al., 2018). This preserves byte-exact semantics operationally: a reference is never allowed to rely on missing chunk bytes merely because metadata exists.

Garbage identification is coupled to the same flag mechanism. CIT entries that remain at Flag=0 past a threshold are collected and removed by a garbage collection thread (Khan et al., 2018). The significance of this approach is that it provides transactional integrity without journaling or write-ahead logs, keeping the dedup write path relatively light while still preserving exact chunk validity (Khan et al., 2018).

Encrypted deduplication introduces a different consistency problem: preserving exact chunk equality under encryption while limiting information leakage. In encrypted systems based on deterministic message-locked encryption, identical plaintext chunks map to identical ciphertext chunks, which preserves byte-exact deduplication but also reveals chunk frequency distributions and locality (Li et al., 2019). The paper on frequency-analysis attacks shows that deterministic encryption preserves plaintext chunk frequencies in ciphertext frequencies and enables inference attacks that exploit chunk locality across backups (Li et al., 2019). In backup workloads, locality-based and size-enhanced attacks can infer a substantial fraction of plaintext chunks when an adversary has access to the ciphertext stream and auxiliary knowledge of prior backups (Li et al., 2019).

Two defenses are proposed: MinHash encryption, which derives keys at segment level rather than per chunk, and scrambling, which breaks local chunk-order correlations within segments (Li et al., 2019). These defenses deliberately weaken perfect byte-exact cross-segment determinism, slightly reducing deduplication efficiency while substantially decreasing inference rates (Li et al., 2019). The implication is not that byte-exact chunk deduplication is incompatible with encryption, but that deterministic equality at chunk level leaks structure by construction. In secure designs, exact deduplication efficiency and confidentiality must therefore be traded against one another rather than treated as independently maximizable goals (Li et al., 2019).

The secure cloud-storage system sec-cs pushes this line further by combining deterministic authenticated encryption with multi-level chunking (Leibenger et al., 2016). Chunks and superchunks are stored in an untrusted key-value store under MAC-derived keys, preserving authenticity and enabling deduplication over encrypted content (Leibenger et al., 2016). Because identical plaintext chunks or superchunks produce identical authenticated ciphertext representations, the system preserves byte-exact deduplication semantics while providing a formal leakage model and integrity guarantees (Leibenger et al., 2016). Its multi-level chunking schemes ML-SC and ML-CDC then address metadata efficiency by deduplicating not only leaf chunks but also superchunks across versions (Leibenger et al., 2016).

6. Performance, asymptotics, and contemporary directions

The practical cost of byte-exact chunk-level deduplication lies in three main areas: chunking throughput, fingerprint computation, and metadata lookup/update. The Ceph cluster-wide system reports that SHA-1 fingerprinting is CPU-intensive and that the overhead is most visible at small chunk sizes and low dedup ratios, where the cost of fingerprinting and redirection is not offset by space savings (Khan et al., 2018). At the same time, its distributed DM-Shard design avoids the scalability collapse of centralized deduplication servers; in the reported experiments, centralized metadata becomes a bottleneck as client thread count rises, whereas distributed metadata allows throughput to scale upward (Khan et al., 2018).

Chunking itself has become a major performance topic. SeqCDC reports that unaccelerated chunking can be the principal bottleneck in CDC pipelines and proposes a hashless, sequence-based CDC algorithm with content-defined skipping and SIMD acceleration (Udayashankar et al., 27 May 2025). Its evaluation reports 15x higher throughput than unaccelerated and 1.2x–1.35x higher throughput than vector-accelerated data chunking algorithms while only minimally affecting space savings (Udayashankar et al., 27 May 2025). VectorCDC generalizes the acceleration of hashless CDC algorithms such as AE, RAM, and MAXP using SSE, AVX, NEON, and VSX, achieving 8.35x–26.2x higher throughput than existing vector-accelerated techniques without affecting deduplication space savings (Udayashankar et al., 7 Aug 2025). These results matter because they shift the bottleneck back toward chunk fingerprinting and storage I/O, making byte-exact CDC-based deduplication more deployable at large chunk sizes and on commodity CPUs (Udayashankar et al., 27 May 2025, Udayashankar et al., 7 Aug 2025).

A 2024 comparative study of CDC algorithms complements these engineering results by distinguishing throughput from deduplication efficacy and chunk-size variance (Gregoriadis et al., 2024). It finds that Gear, especially with normalization, and AE offer strong trade-offs among throughput, deduplication ratio, and chunk-size control, while also uncovering limitations of RAM, PCI, and BFBC under certain workloads (Gregoriadis et al., 2024). This suggests that modern chunker selection for byte-exact deduplication should be workload-specific and multi-objective rather than based on a single historical default such as Rabin.

Theoretical work adds a complementary perspective. The information-theoretic analysis of deduplication shows that fixed-length deduplication can be asymptotically optimal when source blocks align perfectly with chunk boundaries, but can become linearly suboptimal under mild boundary desynchronization (Niesen, 2017). Variable-length and multi-chunk schemes recover much of that loss, with multi-chunk deduplication shown to be order optimal under fairly mild assumptions (Niesen, 2017). Generalized deduplication then extends beyond exact matching by modeling chunks as bases plus deviations, with classic byte-exact chunk deduplication recovered as the special case $\mathcal{Y}=\{\boldsymbol{0}\}$3 (Vestergaard et al., 2019). That paper shows that both classic and generalized schemes converge to near-entropy cost asymptotically, but generalized deduplication can converge much faster when data consists of many small variations around a smaller base set (Vestergaard et al., 2019). This suggests a precise boundary between byte-exact chunk-level deduplication and similarity-based coding: the former is optimal when exact repeats dominate or when simplicity and determinism are required, while the latter is advantageous when data contains many near-duplicates rather than repeated exact chunks (Vestergaard et al., 2019).

Recent work has also extended byte-exact deduplication semantics beyond storage. In Retrieval-Augmented Generation, byte-exact chunk-level deduplication removes duplicate retrieved passages or chat-history records before prompt assembly, preserving only first occurrence order (Schelpe, 11 May 2026, Schelpe, 10 May 2026). Merlin reports sustained speeds of up to 8.7 GB/s and input reduction ranging from 13.9% in low-redundancy datasets to over 71% in high-redundancy pipelines while maintaining absolute data fidelity (Schelpe, 11 May 2026). A companion empirical study measures 0.16% reduction on clean BeIR passages, 24.03% reduction on constructed enterprise patterns, and 80.34% reduction in multi-turn conversational settings, with multi-vendor human evaluation finding no measurable quality regression in the tested RAG regimes (Schelpe, 10 May 2026). This is a newer application domain, but it underscores the portability of the byte-exact chunk-dedup paradigm: wherever inputs are already segmented into stable chunks or records, deterministic exact-match elimination can remove redundancy with minimal semantic risk.

A plausible implication is that the future development of byte-exact chunk-level deduplication will continue along two partially independent axes. One axis concerns systems realization: faster CDC, distributed metadata, robust consistency, secure deployment, and workload-adaptive indexes (Khan et al., 2018, Udayashankar et al., 27 May 2025, Udayashankar et al., 7 Aug 2025). The other concerns scope extension: integrating exact dedup as a conservative first stage before similarity-aware compression, generalized deduplication, or higher-level context optimization (Vestergaard et al., 2019, Schelpe, 11 May 2026). What remains stable across these directions is the defining principle: exact duplication is decided at chunk granularity by deterministic content identity, and the value of the technique derives from preserving that identity across storage, transport, or inference workflows without conflating exact repetition with approximate similarity.