Coding Theorem Method (CTM)

• CTM is an empirical framework for approximating the Kolmogorov complexity of finite objects by simulating small Turing machines based on algorithmic probability.
• It operationalizes the algorithmic coding theorem through exhaustive enumeration and simulation, outperforming traditional lossless compression methods for short, structured data.
• The Block Decomposition Method (BDM) extends CTM to larger, multidimensional objects by aggregating local complexity estimates with logarithmic corrections.

The Coding Theorem Method (CTM) is an empirical framework for approximating the Kolmogorov complexity of finite objects—particularly short strings and small arrays—by leveraging the algorithmic probability concepts introduced independently by Solomonoff and Levin. CTM offers an operationalization of the algorithmic coding theorem via massive-scale enumeration and simulation of small computational devices, typically Turing machines. Its robust generalization for arbitrary-sized or multidimensional objects is realized through the Block Decomposition Method (BDM). CTM and BDM are distinct from, and demonstrably superior to, lossless compression methods for short and structured data, providing provably consistent, local, and global complexity estimates tied directly to algorithmic probability rather than statistical redundancy (Zenil et al., 2016, Gauvrit et al., 2014, Zenil et al., 2012, Leyva-Acosta et al., 30 Jul 2024).

1. Theoretical Foundation

The CTM is anchored in algorithmic information theory, particularly on the notions of Kolmogorov complexity $K(s)$ and algorithmic probability $m(s)$. Let $U$ denote a universal, prefix-free Turing machine. The complexity $K(s)$ is the minimum program length (in bits) for which $U$ halts and outputs $s$; $m(s)$ is the probability that a random program will output $s$ and halt: $m(s) = \sum_{p: U(p)=s} 2^{-|p|}$ Levin’s Coding Theorem relates these by: $K(s) = -\log_2 m(s) + O(1)$ This implies that “simple” strings, produced by many or shorter programs, are more algorithmically probable and thus have lower complexity.

Kolmogorov complexity is uncomputable due to the halting problem; CTM circumvents this by exhaustively enumerating all small programs (e.g., Turing machines with limited states and symbols), empirically constructing $D(s)$—the output frequency distribution—and approximating $K(s)$ by $-\log_2 D(s)$ (Gauvrit et al., 2014, Zenil et al., 2016).

2. Methodology and Computational Workflow

The CTM proceeds through explicit enumeration and simulation of a finite computational model:

• Model Selection: Fix a class of computational devices, e.g., $(t,k)$-space of Turing machines with $t$ states and $k$ symbols, or a high-level, additively optimal prefix machine such as IMP2 (Leyva-Acosta et al., 30 Jul 2024).
• Enumeration and Simulation: Systematically generate all possible devices in this space. For each, simulate its execution on a blank input or tape up to either the Busy Beaver step bound (if known) or a practical resource threshold; record the output if the computation halts.
• Empirical Semi-measure Construction: Count the number of distinct devices that output a given $s$ upon halting. Normalize by the total number of halting devices to obtain

$$D_{t,k}(s) = \frac{|\{T \in (t,k) : T \downarrow s\}|}{|\{T \in (t,k) : T \downarrow\}|}$$

• Complexity Estimation: Assign

$CTM(s) = -\log_2 D_{t,k}(s)$

as an empirical estimate for $K(s)$.

To address the intractability for large $t,k$ or longer outputs, BDM decomposes objects into small sub-blocks for which $CTM$ values are available, and aggregates them with logarithmic penalties for multiplicities (Zenil et al., 2016).

The table below summarizes CTM and BDM’s key computational principles:

Step	1D CTM	Block Decomposition (BDM)
Simulation Target	Small Turing machines on blank tape	All sub-blocks (1D/2D) present in large object
Complexity Assignment	$-\log_2 D_{t,k}(s)$	$\sum_i [CTM(s^i) + \log_2 n_i]$
Scalability	Not scalable to long objects	Scales via local $CTM$ tables, overlap, and tiling
Boundary Handling	N/A	Trim, cyclic, recursive, or padded approaches

Boundary strategies (trim, cyclic, etc.) trade under-or overestimation against computational complexity.

3. Formal Structure and Extensions

CTM has been extended beyond strings to multidimensional objects. For $w$-dimensional arrays (e.g., adjacency matrices or cellular automata spacetime diagrams), decomposition proceeds over subarrays or blocks, and the empirical distribution $D$ is assessed using enumerations of $n$-dimensional Turing machines (Turmites) (Zenil et al., 2012, Zenil et al., 2016).

Block Decomposition adapts as: $$BDM(X) = \sum_{(r_i, n_i) \in Adj(X)} [CTM(r_i) + \log_2 n_i]$$ where $Adj(X)$ enumerates all unique subarrays $r_i$ with multiplicity $n_i$ up to a fixed block size. This approach allows direct application of CTM-derived results for large complex objects, retaining the local nature of algorithmic complexity estimation.

Error bounds and theoretical guarantees have been established. BDM provides upper and lower bounds on $K(X)$, and in the worst-case limit of small block size or highly repetitive data, it smoothly degrades to block Shannon entropy. For sufficiently large and accurate CTM blocks, $BDM \to K$ up to additive error dominated by boundary size and block representation overhead (Zenil et al., 2016).

4. Empirical Validation and Comparisons

Empirical evaluation validates CTM in several domains:

• Compressibility Correlation: Partitioning short strings by their CTM values yields monotonic increases in compressed file sizes under Deflate and other algorithms. Thus, CTM ranks correlate with practical compressibility wherever compression is statistically viable, although CTM offers superior discrimination for very short data (Zenil et al., 2012).
• Complex Sequences: For $\pi$’s digits or the Thue–Morse sequence, CTM and BDM assign significantly lower complexity than block entropy, correctly identifying algorithmic regularity missed entirely by statistical approaches (Zenil et al., 2016).
• Graphs: BDM applied to adjacency matrices distinguishes dual, isomorphic, and cospectral graphs—structures indistinguishable by entropy—demonstrating high structural Spearman correlations ($\rho > 0.9$) (Zenil et al., 2016).
• Alternative Models: CTM performed using high-level, additively optimal languages such as IMP2 remains additively optimal, but may show nontrivial divergence from lower-level CTM approximations on local rank orderings, suggesting slow convergence to the universal distribution for certain high-level grammars (Leyva-Acosta et al., 30 Jul 2024).

5. Implementation, Practicalities, and Parameterization

CTM datasets are precomputed for all short strings over typical $(n,k)$ Turing machine spaces (e.g., $(5,2)$, $(4,9)$) and stored as lookup tables, with effective coverage up to length 12 for binary strings and smaller maximal block size for 2D arrays (e.g., $4\times 4$) (Gauvrit et al., 2014, Zenil et al., 2016). BDM applies CTM locally with user-specified block and overlap parameters; typical ranges are block sizes $l=8\dots12$ for 1D, $3\dots4$ for 2D (Zenil et al., 2016, Zenil et al., 2012). Efficiency depends on the decomposition protocol—non-overlapping BDM is linear, recursive multi-dimensional approaches are polynomial in object size.

Boundary effects are controlled through trimming, cyclic extensions, recursive overlap, or minimal complexity padding, with known error rates (e.g., $O(\mathrm{poly})/\exp(\text{size})$ for trimmings), and theoretical guarantees on upper and lower bounding $K(X)$ (Zenil et al., 2016).

The ACSS R package and several language implementations (Wolfram, Matlab, Python, C++, etc.) provide direct access to CTM and BDM complexity observables for practical use (Gauvrit et al., 2014, Zenil et al., 2016).

6. Scope, Robustness, and Limitations

CTM offers the only currently viable, empirically precise approximation for $K(s)$ in the domain of short, structured, or high-dimensional data, where compression-based methods fail—due to block-coding overhead and file-structure artifacts (Gauvrit et al., 2014). Correlations between CTM distributions computed over different machine parameters (e.g., 3, 4, or 5 states; 1D vs. 2D) are high, with empirical invariance constants small relative to the span of $K(s)$ (Zenil et al., 2012). This provides robustness with respect to model choice and affirms the universality postulated by the invariance theorem.

Limitations primarily arise from exponential scaling—block size, machine class, or program length all rapidly inflate the computational resources required. For high-level models (e.g., IMP2), empirical frequencies show global monotonic correlation with lower-level Turing machine CTM, but significant differences in local fine structure, indicating the need for larger spaces for convergence or alternative grammar choices (Leyva-Acosta et al., 30 Jul 2024).

7. Applications and Future Directions

CTM and BDM have been deployed in a range of scientific disciplines:

• Graph Analysis: Robust discrimination of graph isomorphism classes via adjacency matrix complexity (Zenil et al., 2016).
• Psychology: Objective quantification of perceived randomness in short sequences, correlating with human judgments (Gauvrit et al., 2014).
• Dynamical Systems: Automated categorization of cellular automata and space–time diagrams into known Wolfram classes (Zenil et al., 2012).
• Algorithmic Biology: Applications to molecular folding and network reduction, exploiting the ability to detect non-statistical regularity (Zenil et al., 2012).

Ongoing research centers on the improvement of grammar-encoding strategies for high-level models, more efficient enumeration and pruning (to mitigate the excessive fraction of non-halting or invalid programs), and distributed implementations for further scaling (Leyva-Acosta et al., 30 Jul 2024). The theoretical challenge remains to precisely quantify the additive constant in the coding theorem empirically and to map convergence properties across computational formalisms.

---

References:

(Zenil et al., 2016, Gauvrit et al., 2014, Leyva-Acosta et al., 30 Jul 2024, Zenil et al., 2012)