Alchemy-Random: Meta-RL & Lambda Dynamics

• Alchemy-Random is a framework that integrates meta-RL with procedurally randomized latent causal structures and lambda expression generators to study self-organization.
• It employs a POMDP-based meta-RL benchmark where agents must infer hidden chemistry parameters instead of memorizing policies.
• Its computational models harness random lambda-expression ensembles to simulate chemical reaction networks and reveal emergent autocatalytic behaviors.

Alchemy-Random refers to a class of benchmarks and generator mechanisms underpinning two complementary research domains: meta-reinforcement learning (meta-RL) with procedurally randomized latent structure, and computational models of self-organization based on random $\lambda$-calculus expression ensembles. Across both domains, “randomness” in Alchemy is not a superficial input perturbation but a fundamental feature of (a) the stochastic generation of hidden causal “chemistries” within meta-RL tasks and (b) the ensemble properties and dynamical outcomes produced by different algorithms for sampling random computational objects. As such, Alchemy-Random forms a rigorous cornerstone for analyzing structure inference, hypothesis testing, organizational stability, and the emergence of higher-order adaptive behaviors in artificial agents and symbolic systems (Wang et al., 2021, Mathis et al., 2024).

1. Alchemy-Random in Meta-Reinforcement Learning: Problem Setup

Alchemy, as a meta-RL benchmark, is formulated as a partially observable Markov decision process (POMDP) indexed by a latent episode-specific “chemistry” parameter $Z$. Each $Z$—representing one of $|\Theta|=167\,424$ possible causal structures—is sampled i.i.d. from a known prior $P(Z)$ at the start of each episode and remains fixed across its 10 trials. The structural randomness of $Z$ is crucial: it obliges any agent to infer the functional consequences of actions online rather than memorizing policies.

Key elements:

• State space $S$: Agent pose, stone latent coordinates $c_i \in \{-1,1\}^3$ (for $i=1,2,3$ per trial), and $Z$ (comprising a subgraph $Z$0 of a 3-cube and axis-mapping matrices).
• Action space $Z$1: In the 3D variant, 9 continuous dimensions (e.g., strafe, grasp, turn); in the symbolic variant, discrete actions over stone and destination (potion, cauldron, no-op).
• Transition dynamics $Z$2: Applying potion $Z$3 (effect $Z$4) to stone $Z$5 updates $Z$6 via $Z$7 if the corresponding edge exists in $Z$8; otherwise $Z$9.
• Reward function $Z$0: Only nonzero for a “drop” action, dependent on $Z$1; possible values are $Z$2.

This construction forces agents to operate under epistemic uncertainty, motivating both multi-episode structure learning (meta-learning) and within-episode posterior inference (Wang et al., 2021).

2. Generation and Properties of Latent Structure in Alchemy-Random

Each episode’s chemistry $Z$3 is generated procedurally using a factorizable prior: $Z$4

• $Z$5 (number of “preconditions” for $Z$6).
• $Z$7: Uniform choice from connected subgraphs of the 3-cube consistent with $Z$8.
• $Z$9, $|\Theta|=167\,424$0: Random $|\Theta|=167\,424$1 rotations and reflections.
• $|\Theta|=167\,424$2, $|\Theta|=167\,424$3: Uniform axis permutations and reflections.

This process yields a combinatorially rich landscape of causal structures, with each $|\Theta|=167\,424$4 altering the mapping from latent chemical states and potion effects to observed features and actionable transitions (Wang et al., 2021).

3. Bayes-Optimal Inference and Analysis Tools

Given known $|\Theta|=167\,424$5, a Bayes-optimal “ideal observer” agent maintains a posterior $|\Theta|=167\,424$6 after $|\Theta|=167\,424$7 trials: $|\Theta|=167\,424$8 Action selection in subsequent trials trades off experimentation (entropy reduction in $|\Theta|=167\,424$9) against immediate exploitation. Exhaustive look-ahead in the symbolic version supports calculation of the true Bayes-optimal policy, enabling the establishment of upper performance bounds ($P(Z)$0 episode reward), analytic metric baselines (number of potions, posterior entropy), and statistical model-comparison tests to assess agents’ implicit structural knowledge (e.g., acquisition of opposite-potion pairings) (Wang et al., 2021).

Key diagnostic metrics include:

• Episode reward
• Potions used in trial 1
• Change in potion use (trial 10 minus trial 1)
• Posterior entropy after early and late trials

Model-fitting can distinguish between agents that recognize latent causal invariants ($P(Z)$1: knows opposite-potion pairings) and those that do not ($P(Z)$2) (Wang et al., 2021).

4. Empirical Consequences: RL Agent Performance and the Role of Privileged Information

Deep RL baselines (IMPALA, VMPO with Transformer-XL/LSTM) achieve only $P(Z)$3140–156 reward—comparable to a random heuristic and far below the ideal observer. Both the reduction in potions used for diagnosis/exploitation and the decline in posterior entropy across trials are absent. Performance does not improve when removing motor or sensory complexity (symbolic vs. 3D tasks yield similar results), implying the central bottleneck is latent-state inference and online counterfactual reasoning, not surface-level sensory-motor processing.

Augmentations reveal these bottlenecks specifically: provision of belief-state vectors or ground-truth $P(Z)$4 code at test time nearly recovers the ideal observer’s performance in symbolic Alchemy, and unsupervised auxiliary losses (predicting category counts and $P(Z)$5) substantially improve learning even without privilege, with scores reaching $P(Z)$6260 in 3D. This supports the interpretation that poor inference and representation learning—rather than agent capacity—limits performance on Alchemy-Random (Wang et al., 2021).

5. AlChemy Random Expression Generators and Dynamical Outcomes

In the classic computational AlChemy framework (Fontana, Buss; revisited (Mathis et al., 2024)), “randomness” encompasses the procedures for generating initial pools of $P(Z)$7-expressions, which profoundly shape emergent dynamical organizations.

Generator Definitions

• Original (Probabilistic-Grammar) Generator: Samples expressions recursively with probabilities $P(Z)$8 at depth $P(Z)$9, increasing $Z$0 linearly to force termination. Variable emissions pick bound names with probability $Z$1. Expected expression size is finite and controlled by $Z$2.
• Permutation (Random-Binary-Tree) Generator: Constructs expressions as uniformly random BSTs of size $Z$3 from random permutations. Node arities correspond to application, abstraction, or variable occurrence, with standardization to close free variables.

Structural Statistics

Generator	Expression Size	Mean Depth	Proportion Abstractions	Uniqueness After Dynamics
Original	Variable ($Z$4)	Long chains possible	Controlled by $Z$5	$Z$6
Permutation	Fixed ($Z$7)	$Z$8	20–30% for large $Z$9	$S$0 (fixed point)

In simulation (with $S$1 expressions, $S$2–$S$3 collision steps), the original generator produces heavy-tailed size distributions that support stable, autocatalytic organizations (“L₀/L₁” sets) in 20–40% of runs and robustly maintain hundreds of unique expressions. The permutation generator, by contrast, produces uniformly reactive, balanced trees that collapse rapidly to the trivial identity fixed-point. Theoretical explanations attribute this contrast to differing reduction and reactivity properties, with autocatalytic loops only able to survive in more heterogeneous, grammar-generated populations (Mathis et al., 2024).

6. Formal Connection to Chemical Reaction Network Simulation

Using a simply-typed extension, AlChemy demonstrates the formal capacity to simulate arbitrary transitions in chemical reaction networks (CRNs). For a CRN with species $S$4 and rules $S$5, one constructs $S$6-expressions where:

• Each species $S$7 corresponds to a base type $S$8.
• Each reaction $S$9 is encoded as combinator $c_i \in \{-1,1\}^3$0 of type $c_i \in \{-1,1\}^3$1.
• Multisets (pools) of typed $c_i \in \{-1,1\}^3$2-expressions map to CRN states.

Simulation proceeds via $c_i \in \{-1,1\}^3$3-reductions and collision dynamics, preserving reaction sequences and catalytic regeneration of $c_i \in \{-1,1\}^3$4. The inclusion of Church-pair eliminators ($c_i \in \{-1,1\}^3$5, $c_i \in \{-1,1\}^3$6) allows output selection, formally closing the correspondence between typed combinatorial $c_i \in \{-1,1\}^3$7-expression dynamics and conventional CRN path execution (Mathis et al., 2024).

7. Significance and Implications of Alchemy-Random

Alchemy-Random operationalizes the interplay between episodic structure learning and self-organization in systems governed by high causal uncertainty and stochastic parametrization. In meta-RL, it exposes the inability of current deep RL methods to perform online latent-structure inference without explicit state or auxiliary scaffolding, even in reduced-complexity settings. In computational chemistry, the phenomenon that generator-induced statistical structure can enable or preclude self-maintaining organizations underscores the role of initial conditions in emergent-order studies. The demonstration that typed extensions permit the simulation of all CRNs establishes AlChemy as a universal framework for modeling the computational substrate of self-organization, providing a bridge between abstract computation and chemical dynamics (Wang et al., 2021, Mathis et al., 2024).