Entropy-Normalized Serialization

• Entropy-normalized serialization is a lossless compression technique that represents a message by its lexicographic index among all symbol permutations matching the observed frequencies.
• It employs Combinatorial Entropy Encoding (CEE), an integer-based method that achieves near-optimal compression within a logarithmic overhead without relying on fractional arithmetic.
• The method is practical for text, genomic, and packet header data, though it requires transmitting symbol counts and precomputing multinomial tables for efficient encoding and decoding.

Entropy-normalized serialization refers to a class of lossless data compression techniques in which a complete message is represented by its lexicographic index among all possible symbol permutations with fixed symbol counts, resulting in a bitstream whose length matches the message's Shannon entropy. Combinatorial Entropy Encoding (CEE) is the canonical methodology underpinning entropy-normalized serialization, providing an encoding that is purely integer-based, achieves optimal compression up to a logarithmic term, and eliminates the need for fractional arithmetic or explicit source models. In CEE, the compressed output consists of the lexicographic index (under the multinomial enumeration) plus a vector of symbol frequencies, thus enabling unique and optimal reconstruction of the original message (Siddique, 2017).

1. Formalism and Mathematical Foundations

Let the alphabet $A = \{\alpha_1, \alpha_2, \ldots, \alpha_t\}$, with message $s = s_1 s_2\ldots s_n$ containing $f_i$ copies of $\alpha_i$ such that $\sum_{i=1}^t f_i = n$. The total number of distinct permutations is given by the multinomial coefficient: $$\binom{n}{f_1, f_2, \ldots, f_t} = \frac{n!}{f_1! f_2! \cdots f_t!}$$ Assigning to $s$ its zero-based lexicographic rank $L(s)$ among all such permutations, one obtains the compact index for entropy-normalized serialization. The value $L(s)$ is computed as

$$L(s) = \sum_{i=1}^{n} \sum_{\substack{\beta < s_i \ \beta\in A \ f_\beta > 0}} \binom{n-i}{f_1^{(i)},...,f_{\rm index(\beta)}^{(i)} - 1, ..., f_t^{(i)}}$$

with $f_j^{(i)}$ representing the symbol counts remaining before placing $s_i$.

In the binary case ($A = \{0,1\}$), the calculation simplifies to: $$L(s) = \sum_{i=0}^{n-1} b_i \binom{i}{\#\text{ones seen so far}}$$ where $b_i$ is the $i$th bit, indexed from the least significant position.

2. Encoding and Decoding Algorithms

CEE relies on iterative computation of the lexicographic index for encoding, and the inversion of this process for decoding.

Encoding

1. Initialize $L \gets 0$ and the counter array $\{f_i\}$.
2. For each symbol $s_i$, add for every $\beta < s_i$ with $f_\beta > 0$:

$\binom{n - i}{f_1, ..., f_\beta-1, ..., f_t}$

1. Decrement $f_{s_i}$.
2. Return $(L, \{f_i\})$.

Decoding

1. Initialize as above.
2. For each symbol position, iterate $\alpha \in A$ with $f_\alpha > 0$. Subtract the weighted multinomial term until $L < w$, where $w$ is the term as above.
3. Set $s_i = \alpha$, update $L$ and counts.
4. Continue until all symbols are decoded.

This process involves no multiplications at encode-time and relies on precomputed integer multinomials.

3. Efficiency, Computational Complexity, and Operational Trade-Offs

CEE requires $O(n t)$ lookups per message of length $n$ over an alphabet of size $t$, reducing to $O(n)$ in the binary case. Per-symbol operations are integer additions, subtractions, and array indexing. Space complexity is dominated by storage for factorial or multinomial tables, with precomputation in $O(n^2)$ (or $O(n)$ for Pascal's triangle in the binary case).

Operation	Encoding/Decoding Complexity	Memory (Precomputed Tables)
Arbitrary Alphabet	$O(n t)$	$O(n t)$
Binary Alphabet	$O(n)$	$O(n)$

No explicit entropy model is required, and side information cost ($\simeq t \log_2 n$ bits for $\{f_i\}$) becomes negligible as $n \gg t$. Large alphabet sizes increase decoding cost, because the decoder branches over $t$ symbols per output.

4. Compression Bound and Relation to Shannon Entropy

CEE achieves a theoretical code length of $\lceil \log_2 \binom{n}{f_1,\ldots,f_t} \rceil$ bits for message $s$. By combinatorial analysis,

$\log_2 \binom{n}{f_1,\ldots,f_t} < n H + O(\log n)$

where $H = -\sum_i p_i \log_2 p_i$, $p_i = f_i/n$. The redundancy per symbol vanishes as $n \rightarrow \infty$, guaranteeing asymptotic optimality matching the Shannon entropy. CEE uses strictly fewer than $nH$ bits even for finite $n$, outperforming Huffman and fixed-length codes in such settings.

5. Detailed Worked Examples

Binary Case: For $s = 11010$ with $n = 5$, $f_1 = 3$, $f_0 = 2$, and $b_0$ to $b_4$ indexed from the right:

• $i=0$ ($b_0=0$): no addition, $f_0 \to 1$.
• $i=1$ ($b_1=1$): add $\binom{1}{1} = 1$, $f_1 \to 2$.
• $i=2$ ($b_2=0$): no addition, $f_0 \to 0$.
• $i=3$ ($b_3=1$): add $\binom{3}{2} = 3$, $f_1 \to 1$.
• $i=4$ ($b_4=1$): add $\binom{4}{1} = 4$, $f_1 \to 0$.

Total $L = 1 + 3 + 4 = 8$. The bitstream “1000” and $\{f_1=3, f_0=2\}$ suffice to reconstruct $s$.

Non-binary Case: The message "BANANA" ($n=6$) over $\{A,B,N\}$ with $(f_A=3, f_B=1, f_N=2)$ yields a lexicographic index of 22 among 60 possible permutations. Serialized, this requires $\lceil \log_2 60 \rceil = 6$ bits (e.g., “010110”) plus symbol counts.

6. Comparative Analysis with Huffman and Arithmetic Coding

CEE maps the entire message to a single integer using purely integer operations, contrasting with:

• Huffman Coding: Assigns static, integer-length codewords; suffers inefficiency for non-dyadic distributions and cannot utilize fractional bits.
• Arithmetic Coding: Maps to a fractional interval via repeated multiplication and renormalization, requiring real arithmetic, high precision, or scaled integer arithmetic.

CEE, by contrast, performs only additions and avoids multiplication at encode time. It does not require prior knowledge or estimation of probabilities but operates on observed symbol frequencies per block.

7. Applications, Limitations, and Open Questions

Applications include lossless compression scenarios for small alphabets (such as textual, genomic, or packet header data), embedded or hardware systems lacking floating-point units, and any situation favoring efficient block-adaptive coding.

Limitations are:

• Requirement to transmit symbol counts per block (amortized for large $n$).
• Memory overhead for binomial/multinomial tables when $n$ is large.
• Linear cost in the alphabet size during decoding, impacting scalability for large $t$.
• Not suited to streaming contexts without buffering, as symbol counts must be known per block.

Open problems include unifying the side-information and index streams to remove header redundancy, and adaptation for Markov or context-based sources beyond IID models, suggesting the need for hybrid or hierarchical models atop CEE (Siddique, 2017).