--- title: BigToM Forward Belief Scenarios url: https://www.emergentmind.com/topics/bigtom-forward-belief-scenarios type: topic --- # BigToM Forward Belief Scenarios BigToM Forward Belief Scenarios are a class of systematically controlled theory-of-mind (ToM) reasoning tasks designed to probe first-order belief tracking in artificial agents, particularly large language models (LLMs). These scenarios operationalize the classic false-belief paradigm within computational and neural frameworks, providing a rigorous platform for benchmarking belief attribution, stepwise uncertain inference, and belief dynamics over multi-agent and multimodal environments. BigToM forward belief scenarios have become a key empirical and methodological touchstone for evaluating and developing ToM capabilities in LLMs and related planning systems. ## 1. Formal Definition and Structure of Forward Belief Scenarios A BigToM forward belief scenario consists of a narrative world $W$ and an agent $A$ embedded in a sequence of discrete events $E_1, E_2, \dots, E_k \to E_{k+1}$, where $E_k$ is the last event observed by $A$ and $E_{k+1}$ is a state-changing event not observed by $A$. The agent’s belief at time $t$ is expressed as the subjective probability $$ b_A(t) = \Pr_A[W = W^L \mid \text{Observations}_A \ \text{up to}\ t] $$ At time $k+1$, after an unobserved event causes the true world state to shift from $W_0$ to $W_1\ (\neq W_0)$, the scenario probes: "What does agent $A$ believe about the world?" Formally, the task is to compute $b_A(k+1)$, recognizing that $A$'s belief remains anchored to the last observed state $W_0$ despite the actual world changing to $W_1$ [2511.15895, 2306.15448]. Each BigToM scenario is a tuple: - **Story**: Controlled narrative instantiating abstract causal relations. - **Question**: Queries the agent’s belief after an unobserved event. - **Choices**: Typically binary, e.g. does $A$ believe $p$ or $\neg p$? This framework isolates pure belief state inference and enables tightly controlled experimental conditions over perceptual access and event salience. ## 2. Task Generation via Causal Templates BigToM scenarios are constructed using formal causal templates to specify the syntactic and semantic scaffolding of ToM reasoning [2306.15448]. The main variables are: - $C$: Context (agent, setting) - $D$: Desire (agent’s goal) - $P_0$, $B_0$: Initial percept and derived belief - $E$: Exogenous causal event (world-altering) - $P_1$: Percept of $E$ (or its absence) - $B_1$: Agent’s post-event belief - $A_1$: Subsequent action The causal template induces the following dependency graph: $$ P_0 \rightarrow B_0, \quad E \rightarrow (\text{world state change}), \quad P_1 \rightarrow B_1, \quad (B_1, D) \rightarrow A_1 $$ Concrete BigToM items are synthesized by sampling narrative parameters with LLMs such as GPT-4, enforcing constraints that ensure the intended inference is controlled (e.g., by specifying whether the agent perceives the causal event). This procedural generation supports thousands of evaluation items with high linguistic and situational diversity, matching expert quality according to blinded human raters [2306.15448]. ## 3. Computational Models and Inference Mechanisms Multiple architectures and paradigms implement forward belief reasoning within the BigToM framework: ### a. Scalable Bayesian ToM Planning A stepwise Bayesian update decomposes the belief inference task. At each time step $t$, the agent’s belief $b_t$ is a probability distribution $P(s\mid o_{1:t})$ over hidden states $s$, recursively updated by: - Prediction: $P(b_t\mid o_{1:t-1}) = \sum_{b_{t-1}} P(b_t\mid b_{t-1}, s_t)\,P(b_{t-1}\mid o_{1:t-1})$ - Correction: $P(b_t\mid o_{1:t}) \propto P(o_t\mid b_t)\,P(b_t\mid o_{1:t-1})$ A weak-to-strong control architecture integrates ToM-specialized likelihoods from small LMs with context-rich priors from large LMs, delivering strong generalization and scalability in rich multimodal domains [2506.01301]. #### Example Forward Scenario Calculation | Step | Observation | Updated Belief $b_t$ | |-------|----------------------------|---------------------------------------------------| | $0$ | Uniform over $n$ cabinets | $b_0$ = Uniform($\{cab_1,\dots,cab_n\}$) | | $1$ | $cab_1$ is empty | $b_1$: Zero $cab_1$, re-normalize over $\{cab_2,\dots\}$ | | $2$ | $cab_3$ observation | $b_2$: Sharpened towards $cab_3$ | Over successive steps, the posterior converges to the true location as in a Bayesian filter [2506.01301]. ### b. Regression Approaches Symbolic goal regression reduces the subjective probability $Bel(\phi, do(\alpha,S_0))$ after a sequence of actions and observations to an initial state query $Bel(R[\phi], S_0)$ via recursive term/formula regression operators, sidestepping the need to explicitly unroll the state transition and observation chain [1309.6816]. For ToM and nested beliefs, each additional order introduces another set of regression fluents, allowing collapse to a single initial-state inference. ### c. Feed-forward Belief Propagation In belief networks and neural networks, forward belief propagation analytically tracks marginal means and variances through each network layer for uncertain inputs, using closed-form moment equations adapted for the activation nonlinearity (e.g., sigmoid, ReLU). This provides a differentiable and scalable means of maintaining uncertainty during forward passes, necessary for efficient ToM architectures based on deep networks [1803.10590]. ## 4. Evaluation, Performance Benchmarks, and Error Profiles BigToM forward belief scenarios are central to recent ToM evaluation efforts. Key findings include: - **Human baselines**: ≈100% (True-Belief), ≈92% (False-Belief), combined ≈92% accuracy. - **GPT-4**: Near-ceiling performance in 0-shot, with forward belief (false-belief) accuracies .98–.99, but with a mild anchor bias when the agent's initial belief is reiterated (.97→.90) [2306.15448]. - **Smaller LLMs**: Dramatically lower, e.g., LLaMA-65B: .41, text-davinci-003: .25, gpt-3.5-turbo: .31, Claude-v1.3: .59, Claude-2: .52. - **Contrastive Activation Addition (CAA) steering**: Applied to Gemma-3-4B, CAA increased accuracy from 32.5% to 46.7% on 1,000 forward-belief scenarios, an absolute gain of 14.2% [2511.15895]. - **Hybrid planners (e.g., 8B→405B chain)**: Achieve 81.3% on multimodal benchmarks, with strong generalization to unseen domains and +4.6% improvement over prior state-of-the-art [2506.01301]. Weaker models typically fail to track perceptual access, defaulting to world-state reasoning (what is true) rather than the agent’s belief state (what they think), or they anchor erroneously on initial beliefs. ## 5. Planning and Symbolic Formulations Belief-space planners leverage planning graph heuristics adapted for progression in uncertain, partially observable domains [1103.1711]. Key constructs include: - Belief state distances ($d(BS, BS')$) via aggregation (max, sum, overlap). - Relaxed plan or level-heuristics computed via single, multiple, or labelled-uncertainty graphs (LUGs), with tracking of variable dependencies in BDD data structures. - Integration into A*/AO*-style planners allows scalable search and pruning in large belief spaces relevant for robotic and social ToM tasks. Such methods generalize to hierarchical or nested ToM settings in BigToM, provided variables (actions, observations) are suitably factored or extended to account for higher-order beliefs. ## 6. Mechanistic Insights from Neural ToM Studies Recent studies decompose LLM ToM improvements in forward belief scenarios by analyzing internal activations using linear probes across 45 cognitive actions [2511.15895]. Contrastive activation steering (CAA) was shown to: - **Enhance emotion-related activity**: emotion perception (+2.23 layers), emotion valuing (+2.20 layers). - **Suppress analytical routines**: questioning (-0.78), convergent thinking (-1.59). The principal mediator of improved ToM performance in LLMs is increased engagement of emotional processing circuitry rather than traditional analytic reasoning. This challenges the assumption that logical chain-of-thought is the key path to social inference in LLMs; perspective-taking arises from affective mechanisms. ## 7. Belief Maintenance and Scenario Management Systems Belief Maintenance Systems (BMS) generalize classical Truth Maintenance Systems (TMS) to infinite-valued logic, allowing for dynamic and reversible assertion of hypothetical facts and propagation of degree-of-belief [1304.3084]. In the context of BigToM-style scenarios, BMS: - Represents each fact as a node with $(b^+, b^-)$ belief degrees. - Utilizes invertible Dempster–Shafer mass assignments to support dynamic forward scenario evaluation ("what if $X=1$"). - Allows rollback to previous belief states by removing assertions and repropagating support. This infrastructure enables fine-grained, reversible scenario analysis in forward belief tasks, providing a foundation for rich what-if explorations in ToM and planning. --- BigToM forward belief scenarios operationalize first-order belief reasoning in computational contexts, offering a rigorous benchmark for ToM, planning, and neural inference methods. Their systematic structure, empirical tractability, and extensibility to multimodal and hierarchical settings make them central to ongoing progress in machine social cognition. Recent advances clarify both the statistical and mechanistic pathways underlying LLM ToM abilities and provide a roadmap for future research into scalable, robust, and transparent belief inference systems. Source: https://www.emergentmind.com/topics/bigtom-forward-belief-scenarios