Learning Explicit Behavioral Models with Adaptive Questions and World-Model Probes

Abstract: Interactive agents trained only against task return can achieve high scores while failing to represent the mechanisms that make their actions succeed. This makes brittle behavior difficult to diagnose and limits adaptation when environment dynamics change. Existing LLM reflection and policy-code repair can revise behavior from failed trajectories, but questions and world-understanding tests are usually used only after training. We introduce an Explicit Symbolic Behavioral Model (ESBM), a trainable behavioral model that couples task performance with evidence-grounded question answering and executable mechanism prediction. An ESBM represents behavior through typed predicates, weighted rules, bounded options and mechanism memory; the mechanism layer predicts symbolic events, object changes, rewards and terminal consequences under action interventions. After each rollout, adaptive questions and active world-model probes convert score failures, QA errors and transition-prediction errors into constraints for local ESBM edits. Candidate models are selected by a multi-criterion rule that jointly evaluates task score, answerability and active world-model consistency. Under the tested Atari-style protocols, ESBM learns high-scoring policies while producing explicit answers and executable mechanism predictions, indicating that adaptive questions can serve as both training pressure and reusable benchmarks for mechanistic policy learning in this setting.

• The paper introduces the ESBM framework that combines explicit symbolic representations with reinforcement learning to address shortcut behaviors and mechanism shifts.
• It employs a Challenger-Optimizer architecture to generate adaptive QA and counterfactual world-model probes for precise, evidence-based mechanistic repairs.
• Experimental results on JAXAtari games demonstrate ESBM’s superior adaptation, higher testable understanding, and recovery in nonstationary environments compared to conventional RL models.

Explicit Symbolic Behavioral Model (ESBM): Optimizing Interactive Agents with Adaptive Mechanistic Probing

Motivation and Problem Definition

Conventional interactive agents in RL, particularly those optimized for task return, are frequently susceptible to mechanism-agnostic “shortcut” behavior—achieving high aggregate rewards while lacking any testable representation of the mechanisms dictating environment dynamics. This phenomenon results in brittle policies with minimal generalization and recoverability when underlying dynamics shift. The ESBM framework addresses this failure mode by augmenting reward optimization with explicit symbolic representations of mechanism knowledge, constructed to be directly queryable, executable, and locally editable.

Figure 1: The distinction between high task return and mechanistic understanding: reward-optimized policies can fail catastrophically under mechanism shifts, while ESBM exposes and repairs such failures through explicit predicates, rules, and mechanistic edits.

Architecture and Training Loop

The ESBM is a rich symbolic model comprising a vocabulary $\Sigma$, state abstraction predicates $\Phi$, weighted policy rules $\Pi$, a bounded option library $O$, and executable mechanism memory $K$. Execution decomposes into symbolic abstraction, rule-based action selection, option invocation, and mechanistic state prediction. Unlike LLM-based reflection or code-edit agents, the LLM backend in ESBM proposes only local, type-checked edits, which are accepted if validated against a joint criterion including reward, QA accuracy, and world-model consistency.

A Challenger-Optimizer architecture underpins the learning dynamics. The Challenger leverages recent rollouts, Q&A mistakes, symbolic diffs, transition-prediction errors, and mechanism uncertainty to generate policy-adaptive QA and interventional world-model probes. The Optimizer proposes edits to ESBM which are filtered by a verifier gate to prevent regression and preserve protected behavioral constraints.

Figure 2: ESBM training loop—Challenger generates adaptive probes from model failures, Optimizer proposes typed symbolic repairs, Verifier ensures only validated ESBM updates are accepted.

Symbolic and Mechanistic Model Design

The ESBM diverges from classical symbolic policies by fusing interpretable abstraction with executable mechanism memory. State abstractions ($\Phi$) consist of parameterized, type-checked predicates mapping raw observations to semantic facts. The rule layer ($\Pi$) realizes action selection with bodies referencing abstractions and heads as primitive actions or bounded options. Option contracts ($O$) express local procedures explicitly delimited by preconditions, termination, and resource invariants.

Mechanism memory ($K$) is a first-class, auditable data structure that, for each symbolic state and action sequence, predicts resultant facts, object changes, rewards, and terminal events, supporting both QA and simulation probes. Edits to any ESBM submodule are validated as concrete diffs, enabling precise diagnosis and repair.

Policy-Adaptive QA and Active Probing

A central innovation is integrating policy-adaptive, evidence-grounded QA and active counterfactual probes directly into learning. The Challenger adaptively generates:

• Grounded QA: Target entities, hazards, strategies, and game mechanics, emphasizing areas recently associated with model failures or low confidence.
• Active world-model probes: Interventional rollouts from matched checkpoints, comparing the consequence predictions in $K$ with environment rollouts, thus stress-testing mechanistic veracity.

Answer correctness is graded not only by match to gold but also by whether it is anchored in explicit model evidence (predicates, rules, options, mechanism entries, or trajectories). Similarly, transition predictions are validated both passively (on held-out data) and actively (counterfactual interventional branches).

Multi-Criterion Model Selection and Local Learning

Each candidate ESBM edit proposes not only to enhance reward, but must also clear thresholds in QA accuracy and (passive and active) world-model prediction. The vectorized acceptance rule enables tolerable tradeoffs (improvement in one validated signal tolerated as long as regressions are within tight bounds for protected aspects). This vector acceptance gate favors mechanistic repairs even if immediate reward does not reflect the value of such changes—a principle crucial for adaptation to future mechanism shifts.

Experimental Analysis

Experiments are conducted on JAXAtari variants of Kangaroo, Seaquest, and KingKong, chosen for their intricate mechanistic dependencies in route planning, hazards, resource management, and action timing.

Task Performance: ESBM surpasses DQN, PPO, NUDGE, and BlendRL in game scores on all three games. For instance, ESBM achieves mean scores of $\Phi$0 (Kangaroo), $\Phi$1 (Seaquest), and $\Phi$2 (KingKong)—outperforming both reward-only symbolic evolution (EvoSymbol) and neuro-symbolic references.

Mechanistic Understanding (Evidence-linked QA): The ESBM maintains high evidence-supported QA accuracy: 0.949 (Kangaroo), 0.732 (Seaquest), 0.855 (KingKong), indicating the learned models possess explicit, auditable evidence supporting correct answers to challenging mechanistic questions.

Executable World Model: ESBM’s mechanism memory strongly dominates persistence, constant-dynamics, and decision-tree symbolic baselines in per-event F1 (0.84 for Kangaroo, 0.80 for Seaquest, 0.74 for KingKong), changed-field prediction, rewards, hazards, and especially on active counterfactual branches (e.g., 0.61 for Kangaroo), demonstrating robust causal abstraction and generalization.

Mechanism-Change Recovery: When environment dynamics are altered (e.g., speed-up monkeys, faster oxygen decay, or bombs), ESBM policies adapt faster and more completely than PPO, recovering a larger fraction of original performance (up to 0.94 in Kangaroo post-shift, compared to PPO’s 0.66). This underscores the essential role of explicit mechanism memory for rapid adaptation in nonstationary environments.

Implications and Limitations

This work formalizes a rigorous approach to integrating mechanistic question answering and executable symbolic world modeling with classic reinforcement optimization paradigms. Practically, this enables deployable agents with transparent mechanisms, rapid post-deployment adaptation, and policy diagnosis. Theoretically, the results challenge the sufficiency of reward-only or latent neural representations for reliable, auditable agent intelligence.

However, the expressivity and robustness of ESBM depend on the coverage and fidelity of the symbolic abstraction pipeline, the quality and diversity of post-hoc QA/probe generation, and the limited scope of Atari-style testbeds. Generator-answerer leakage and the risk of brittle overfitting to QA artifacts remain open methodological concerns. Furthermore, the transition from constrained environments to complex real-world settings will require semi-automatic vocabulary construction, richer predicate templates, and scalable symbolic introspection protocols.

Future Directions

Future work can extend the ESBM approach to robotics and open-ended embodied environments, integrating vision pipelines for automatic symbol grounding, expanding the option interface toward hierarchical skills, and investigating scaling laws for mechanistic abstraction in large RL and planning benchmarks. Cross-comparison with purely inductive structural RL as well as explicit program synthesis approaches is warranted to delineate the precise advantages of typed, auditable mechanism memory.

Conclusion

The ESBM framework, rooted in explicit mechanistic modeling and validated through adaptive QA and executable counterfactuals, establishes a compelling paradigm for interpretable, adaptable agent learning. Its architecture demonstrates that symbolic policy models, when augmented with active probing and strict verifier gates, not only match or exceed state-of-the-art reward-optimized models in score, but far exceed them in testable understanding and resilience to causal shifts in environment dynamics.