SuperARC Testing Framework

• SuperARC Testing Framework is an open-ended, agnostic benchmark that leverages algorithmic complexity and semicomputable approximations (CTM and BDM) to evaluate both artificial and natural agents.
• The framework implements a novel methodology by converting data compression performance into quantitative intelligence scores, uniquely emphasizing the equivalence of prediction and compression.
• Empirical evaluations show that SuperARC achieves near-perfect performance on tailored benchmarks, exposing limitations of conventional LLMs in generating nontrivial, generalizable solutions.

The SuperARC Testing Framework is an open-ended, agnostic, and data-type–independent benchmark for evaluating the intelligence of artificial and natural agents. Grounded in Algorithmic Information Theory, SuperARC provides quantitative measures resistant to benchmark contamination and memorization, targeting intelligence in narrow AI, AGI, and ASI contexts. Unlike conventional benchmarks, SuperARC leverages semicomputable approximations to Kolmogorov complexity—by methods such as the Coding Theorem Method (CTM) and Block Decomposition Method (BDM)—and eschews reliance on statistical compression techniques and surface-level pattern-matching (Hernández-Espinosa et al., 20 Mar 2025).

1. Theoretical Underpinnings

SuperARC’s foundation rests on two central constructs: Kolmogorov complexity and algorithmic probability. Kolmogorov complexity, $K(s)$, for a string $s$ with respect to a universal prefix Turing machine $U$, is the minimum length $|p|$ of program $p$ such that $U(p) = s$. Algorithmic probability or Solomonoff’s universal distribution, $m(s) = \sum \{ 2^{-|p|} : U(p)=s \}$, is tightly related to $K(s)$ by $m(s) \approx 2^{-K(s)}$ up to an additive constant.

A fundamental equivalence underpins the SuperARC design: the ability to better compress data is equivalent and directly proportional to the ability to predict data asymptotically. The Levin–Schnorr theorem formalizes this for infinite binary sequences $x$; such $s$0 is considered algorithmically random (incompressible) if and only if no computable martingale (effective betting strategy) can achieve unbounded capital growth on $s$1. The construction $s$2 yields a left-semicomputable supermartingale, and the converse establishes that success in compression implies prediction, and vice versa.

2. Architecture and Workflow

The SuperARC framework offers a practical and systematic methodology for intelligence assessment based on recursive compression (abstraction) and inversion of mechanistic models (Bayesian abduction or predictive ‘planning’). Unlike classical statistical compressors (e.g., GZIP, LZW), which approximate Shannon entropy and are limited to surface redundancy, SuperARC operates at the deeper, semicomputable level of algorithmic complexity. This distinction is central to addressing benchmark leakage and resistance to memorization, especially salient in evaluating LLMs and similar agents.

Block Decomposition Method and Coding Theorem Method

Inputs (sequences or more general combinatorial objects) are partitioned into smaller blocks, $s$3. For each block, CTM approximates Kolmogorov complexity via precomputed tables of small Turing-machine outputs. BDM is then calculated as:

$s$4

where $s$5 counts the number of occurrences of block $s$6. This hybrid metric integrates algorithmic and statistical aspects.

Algorithmic Workflow

For datasets at low, medium, and high complexity ($s$7), elements are encoded (e.g., UTF-8 to binary) and each is presented to an agent. The agent must return a program or formula that reproduces the element. Responses are assessed for exactness and complexity (often via BDM), and auxiliary features (such as code length and zlib-compressed size) are computed. The scoring procedure is applied as described in the next section.

3. Scoring Intelligence

Scoring in SuperARC reflects both correctness and non-triviality of agent outputs.

• $s$8 Fraction of correct and non-trivial (i.e., genuine compression model) responses.
• $s$9 Fraction of correct but ordinal mapping responses.
• $U$0 Fraction of correct but print-only responses.
• $U$1 Incorrect responses.

Each solution class receives a compression gain, $U$2, computed as the normalized gain:

$U$3

The final composite intelligence score is:

$U$4

High scores ( $U$5 ) are only achievable by agents that output concise, generalizable mechanistic models; trivial or print-only solutions (and errors) result in low $U$6.

4. Guarantees and Bounds

The SuperARC framework inherits universal intelligence guarantees from theoretical results: in the asymptotic resource-unbounded regime, Levin’s universal search and Solomonoff induction ensure that algorithmic probability–guided search is universally optimal for computable induction problems. The CTM/BDM neurosymbolic pipeline is a practical, resource-bounded instantiation of this formalism. Notably, in the worst case, BDM reduces to a Shannon-entropy–like compressor, guaranteeing that its performance is never inferior to classical statistical compressors. Upper bounds are established: $U$7, with CTM guaranteeing optimality up to a constant:

$U$8

5. Empirical Results

SuperARC’s empirical evaluation comprises two main tracks: short binary sequence prediction and free-form code generation tasks.

Binary Sequences

• Test sets include "climber" strings (low Kolmogorov complexity, nontrivial length) and random binary strings of length 11–20.
• Time-series–oriented LLMs (Chronos, TimeGPT-1, Lag-Llama) and traditional ZIP/LZW compressors are compared.
• Metrics: exact final-digit prediction, set membership, and Levenshtein distance.
• Performance: On climber strings, the best LLM achieves ~70% accuracy, with others at ~50%; for random strings, all systems are at chance (~50%). BDM/CTM achieves near-perfect performance by synthesizing mechanistic generative programs.

Free-Form and Code Generation

When LLMs (GPT-4o, Claude 3.5, Grok, etc.) are prompted to generate formulas or code that reproduce integer sequences of varying algorithmic complexity, correct responses generally degrade into "print(sequence)" constructs or trivial index-based lookups. As complexity escalates, LLMs’ abilities to deliver nontrivial solutions deteriorate, whereas BDM/CTM continues to construct concise, generalizable program families, achieving $U$9.

Agent/Method	Climber Accuracy	Random Accuracy	Nontrivial Code Gen
Best LLM	≈ 70%	≈ 50%	Fails as complexity increases
ZIP/LZW	≈ 50%	≈ 50%	Not applicable
BDM/CTM	≈ 100%	≈ 100% (by design)	Succeeds

6. Implications and Limitations

SuperARC analysis elucidates fundamental limitations of current LLMs. While LLMs excel at memorization and pattern-matching, they generally fail to uncover and abstract causal or mechanistic models required for model creation or planning. This deficiency explains phenomena such as “hallucinations,” which arise from lacking internal causal representations. SuperARC’s open-endedness and grounding in algorithmic probability permit the evaluation of genuine model-building and generalization capabilities, exposing the boundary between mere mastery of surface features and true synthesis.

Limitations include the computational cost of CTM (making it applicable only to small blocks), and the scaling challenges for BDM on large or multimodal data. Furthermore, the choice of encoding and complexity metric introduces bounded additive constants (per the Information Non-Increase Theorem), but these do not impact asymptotic assessment. Ongoing challenges or open research directions include dynamic benchmark generation, expanding Bayesian model priors, and hybrid pipelines that combine LLMs with BDM for tractable yet robust performance (Hernández-Espinosa et al., 20 Mar 2025).

7. Significance and Future Directions

SuperARC establishes a first-principles pathway for robust, contamination-resistant intelligence benchmarking across agent modalities and data types. By tying the assessment of intelligence to the fundamental equivalence of compression and prediction limits, it provides a semicomputable, extensible benchmark to gauge the progress and remaining gaps in the pursuit of universal or superintelligent agents. Prospective future work includes efficient approximations of CTM/BDM to handle larger inputs, dynamic and adaptive test construction, and integration with neurosymbolic or hybrid LLM+BDM systems for improved tractability and generalization potential. Such directions reflect the evolving landscape of intelligence assessment and the search for rigorous, leakage-resistant, and universally meaningful standards.