EsoLang-Bench: Evaluating Esoteric Reasoning

• The paper introduces EsoLang-Bench, a contamination-resistant benchmark that tests algorithmic reasoning through code generation in five obscure, low-resource programming languages.
• It employs multiple evaluation strategies, including zero-shot, few-shot, and agentic methods with interpreter feedback, to assess LLMs' capability to internalize new syntactic and semantic structures.
• Empirical results reveal a stark performance gap beyond easy tasks, highlighting that even advanced models struggle to generalize to medium and harder problems, underscoring genuine reasoning limitations.

EsoLang-Bench is a benchmark suite specifically designed to evaluate genuine algorithmic reasoning capabilities in LLMs by requiring code generation in five esoteric programming languages mostly absent from LLM pre-training corpora. By exploiting both the syntactic and semantic unfamiliarity of these languages and the economic irrationality of their pre-training inclusion, EsoLang-Bench provides robust, contamination-resistant measures of reasoning in scenarios where performance gains cannot be attributed to memorization or data leakage. It is the first evaluation to test LLMs’ capacity to internalize new language formalisms via explicit documentation, interpreter feedback, and iterative experimentation—paralleling human acquisition of exotic programming skills (Sharma et al., 10 Mar 2026).

1. Motivations and Conceptual Foundations

Mainstream code generation benchmarks such as HumanEval and MBPP have observed near-ceiling performance (85–95%) from state-of-the-art LLMs. Such results are increasingly attributed to rote memorization and data contamination. EsoLang-Bench addresses this by selecting “esoteric” programming languages—Brainfuck, Befunge-98, Whitespace, Unlambda, and Shakespeare—that each possess between 1,000× and 100,000× fewer public code repositories than a language like Python. The practical absence of these esolangs from LLM pre-training data renders “benchmark gaming” economically irrational, ensuring that model performance cannot be attributed to retrieval from training corpora.

These esolangs remain Turing-complete and leverage fundamental computational primitives (loops, state, recursion), but they do so with radically distinct, often intentionally obscure or orthogonal syntaxes and semantics. EsoLang-Bench aims to measure genuine transferable reasoning: acquisition of language semantics via documentation, exploration, and interpreter-driven debugging, rather than surface-level pattern completion or latent knowledge of high-resource paradigms.

2. Esoteric Languages and Reasoning Challenges

EsoLang-Bench evaluates five esoteric languages, each imposing distinct reasoning and representation barriers:

Language	Key Features	Unique Challenges
Brainfuck	Eight commands (>,<,+,−,[,],.,,) with memory tape	Manual pointer arithmetic; no variable names
Befunge-98	2D instruction grid; self-modifying code; stack-based	Spatial reasoning in 2D; non-linear control
Whitespace	Only whitespace is syntactic; invisible instructions	Tokenization/encoding; unlearned syntax
Unlambda	Pure combinatory logic; (s, k, i); no variables	Church numerals; deep combinatorial reasoning
Shakespeare	Programs as plays; natural-language encodings	Masked semantics; complex mapping to control

Brainfuck demands precise memory manipulation and low-level loop invariants without named variables. Befunge-98 requires spatial (2D) control flow reasoning and supports instruction pointer directionality. Whitespace’s semantics are encoded in spaces, tabs, and newlines, exposing LLM tokenizers to severe OOD inputs. Unlambda, rooted in combinatory logic, forces the construction of all operations from s, k, i combinators and application, with even arithmetic requiring Church encodings. Shakespeare appears in a readable, natural-language style but operates with an alien semantic mapping.

3. Benchmark Composition and Problem Stratification

EsoLang-Bench contains 80 programming problems transposed into each of the five esoteric languages, yielding 400 evaluation items. Problems are binned by difficulty into four tiers:

• Easy: Single-loop logic or elementary I/O
• Medium: Multiple loops or simple recursion
• Hard: Manipulation of nested data structures or non-trivial string/number algorithms
• Extra-Hard: Advanced algorithms with complex state (e.g., longest increasing subsequence, Josephus problem)

Each benchmarked problem is specified in natural language and accompanied by six input-output examples. Problems are calibrated against their corresponding Python reference solutions and are designed to isolate pure algorithmic reasoning, excluding reliance on specialized libraries or broader programming knowledge.

4. Evaluation Paradigms and Experimental Methodology

Five advanced LLMs (GPT-5.2, O4-mini-high, Gemini 3 Pro, Qwen3-235B, Kimi K2) and two agentic systems (OpenAI Codex, Claude Code) are evaluated across multiple prompting and inference strategies:

• Zero-Shot: System prompt plus documentation, problem statement, and cases; no exemplars.
• Few-Shot (3 exemplars): Above, plus three solved esolang examples demonstrating basic constructs.
• Self-Scaffolding: Iterative code generation, interpreter execution, and automatic feedback loop (up to five iterations).
• Textual Self-Scaffolding: Separate coder and critic agents; critic provides natural-language debugging feedback consumed by the coder.
• ReAct Pipeline: Sequential planner (pseudocode generation), editor (translation to esolang), and critic (error feedback).

Agentic systems leverage interpreter access, dynamic context, and external tool calls (e.g., documentation retrieval). Experiments employ temperature 0.7, three seeds, and bootstrap-derived 95% confidence intervals. Success is recorded only if all six test cases pass exactly, and significance is tested with Bonferroni-corrected Wilcoxon tests.

5. Empirical Outcomes and Capability Gap Analysis

Standard code LLMs achieve 85–95% overall accuracy on Python-based HumanEval/MBPP benchmarks, but only 0–11% on EsoLang-Bench. Define accuracy and capability gap formally as:

$$\mathrm{Accuracy} = \frac{\#\text{problems solved}}{\#\text{problems total}}$$

$$\Delta_L = \mathrm{Accuracy}_{\text{Python}} - \mathrm{Accuracy}_{L}$$

For GPT-5.2 generating Brainfuck:

$$\Delta_{\text{Brainfuck}} \approx 90\% - 6.2\% = 83.8\%$$

Summary of best non-agentic results per language:

Language	Best Accuracy (%)	Highest Tier Solved
Brainfuck	6.2	Easy only (5/20)
Befunge-98	11.2	Easy only (9/20)
Whitespace	0	None
Unlambda	1.2	Easy only (1/20)
Shakespeare	2.5	Easy only (2/20)

All models failed Medium, Hard, and Extra-Hard tasks across the board.

Agentic architectures (interpreter-in-loop) approximately double static baseline accuracy but remain below 15%:

System	Brainfuck (%)	Befunge-98 (%)	Average (%)
Best non-agentic	6.2	11.2	8.7
Codex (Agentic)	13.8	8.8	11.2
Claude Code (Agentic)	12.5	8.8	10.6

This suggests interpreter-access and persistent memory support meaningful improvement, but ultimate performance remains very low.

6. Failure Analysis and Interpretative Synthesis

All evaluated systems exhibited a pronounced “difficulty cliff” at the transition from Easy to Medium: no model solved any Medium, Hard, or Extra-Hard task in any esolang. This pattern reflects the absence of generalizable algorithmic reasoning transferable to novel, out-of-distribution syntaxes.

Error analysis reveals two distinct regimes:

• For Brainfuck and Befunge-98 (rare but nonzero training presence), models display lower syntactic (15–25%) but high logic error rates (35–60%), indicating partial syntax internalization without functional semantics.
• For Whitespace and Unlambda, near-100% compile errors indicate total lack of syntactic familiarity, likely worsened by byte-pair encoding tokenization processes stripping salient information.

Few-shot prompting offers a statistically insignificant accuracy boost over zero-shot (+0.8pp, $p = 0.505$), supporting the interpretation that in-context learning only exploits pre-trained structural priors. Self-Scaffolding yields the highest non-agentic gains, but textual critics are ineffective in such OOD settings due to compounded ignorance.

The extreme scarcity of esoteric language code in available corpora and the economic irrationality of their inclusion prevents contamination. This indicates observed failures stem from authentic reasoning limitations, not benchmark leakage.

7. Synthesis, Implications, and Future Directions

EsoLang-Bench demonstrates that current LLMs, including those with architecturally sophisticated and interpreter-in-the-loop (“agentic”) operation, cannot generalize computational primitives to new language formalisms absent from their pre-training distribution. Techniques such as few-shot prompting and chain-of-thought fail to teach new formalisms, instead amplifying biases from the pre-training data.

Recommendations for advancing the field include expanding coverage to further esoteric paradigms (Malbolge, INTERCAL, Piet), establishing an official, held-out problem leaderboard to combat overfitting, and systematically exploring the compute–accuracy frontier for out-of-distribution tasks. Addressing tokenizer and architectural challenges—particularly to accommodate “invisible” or exotic syntaxes such as Whitespace—is a key avenue for improvement.

By providing rigorous, interpreter-verified, and contamination-resistant assessment, EsoLang-Bench clarifies the current boundaries of LLM algorithmic reasoning and offers a benchmark for evaluating future progress in this domain (Sharma et al., 10 Mar 2026).