Cross-Template Misconception Prediction

• The paper introduces a framework that infers a student’s malrule from a given error and predicts responses across different problem templates.
• Methodologies like MalruleLib and EDGE leverage dual-path trace generation and Bayesian inference to operationalize cross-template diagnostic predictions.
• Results demonstrate a 10–21 point accuracy decline in cross-template settings, underscoring the importance of explicit procedural supervision in adaptive learning.

Cross-template misconception prediction is the task of inferring a student's underlying misconception (malrule) from their systematic error on one problem template and then accurately predicting their misconception-driven response when presented with a novel, distinct problem template governed by the same malrule. This task operationalizes the challenge of procedural generalization: determining whether a diagnostic system can recognize cognitively coherent error patterns and forecast their implications across diverse mathematical contexts, such as algebraic manipulations, measurement, or word problems. The field synthesizes learning-science research on malrule modeling, cognitive diagnostics, psychometrics, and recent advances in large-scale automated reasoning and adaptive learning frameworks (Chen et al., 6 Jan 2026, Verma, 10 Aug 2025).

1. Formalization of Cross-Template Misconception Prediction

At its core, the cross-template prediction problem is formalized as assessing the Malrule Reasoning Accuracy (MRA). Given one instance of a student applying a malrule $m \in \mathcal{M}$ on a template $t_1 \in \mathcal{T}_m$ (with observed trace $S_m(i_s)$), the objective is twofold: (1) infer the correct malrule $\hat{m}$, and (2) predict—given a new instance $i_t$ sampled from a distinct template $t_2 \in \mathcal{T}_m$—the answer $a_m(i_t)$ that the student would produce by following the same misconception. The probability that both steps are correct defines MRA: $$\mathrm{MRA} = \Pr\bigl(\hat{m}=m\;\wedge\;\hat{a}_{\mathrm{next}}=a_m(i_t)\;|\;T_{\mathrm{train}}=t_1, S_{\mathrm{mistake}}=S_m(i_s)\bigr)$$ Evaluation distinguishes same-template (where $t_1=t_2$) from cross-template ($t_1 \neq t_2$) performance, as well as answer-only versus answer+steps prompt formats (Chen et al., 6 Jan 2026).

2. Frameworks: MalruleLib and EDGE

Two complementary frameworks provide infrastructure and theory for this task:

MalruleLib

MalruleLib encodes 101 real student misconceptions as executable Python modules (malrules), each defining parameterized templates ($\mathcal{T}_m$), correct and malrule-driven step traces, and instance generators. This design allows scalable generation of over one million paired (correct, malrule-consistent) step traces across 498 diverse templates, supporting controlled evaluation of both reasoning and cross-template prediction (Chen et al., 6 Jan 2026).

EDGE

EDGE offers a unified adaptive learning pipeline consisting of four stages—Evaluate (ability estimation), Diagnose (Bayesian inference of misconceptions via Dirichlet-mixture posteriors on response patterns), Generate (contrastive counterfactual item synthesis targeting minimal-perturbation to invalidate shortcuts), and Exercise (restless-bandit index scheduling). Shared misconception embeddings across templates and theoretical guarantees on posterior reductions formalize cross-template generalization and remediation (Verma, 10 Aug 2025).

3. Methodologies and Experimental Setup

MalruleLib conducts systematic experiments with nine LLMs, including models from 4B to 120B parameters (e.g., gpt-oss-20b, Phi-4, Llama-3.3-70B, Qwen3-80B). The experimental design varies:

• Prompting: "answer-only" (source mistake only) vs "with-steps" (malrule-driven step trace supplied)
• MRA conditions: same-template (source and target instances from the same template) and cross-template (from different templates)
• Additional "forward MRA" metric: apply a natural language description of the malrule to predict the malrule-driven answer on a single problem

In EDGE, cross-template prediction emerges from the clustering of misconception response patterns (features concatenating item, distractor, time/confidence), and posterior propagation as templates diversify. Counterfactual generation enforces template diversity while psychometric bands and shortcut-invalidation constraints ensure content validity and target the misconception with minimal confounds (Verma, 10 Aug 2025).

4. Key Results: Performance and Generalization Gaps

Empirical results from MalruleLib establish several robust trends:

Metric/Condition	Accuracy (%)
Correct Reasoning (CRA)	65.7
Same-template MRA (answer-only)	56.1
Cross-template MRA (answer-only)	40.5
Cross-template MRA (with-steps)	46.5
Forward MRA (described rule)	32.3

Key findings:

• Cross-template prediction exhibits a 10–21 point accuracy degradation relative to same-template prediction across all models, confirming the challenge of abstract procedural generalization.
• Cross-template drop relative to correct reasoning was 25.3 points (answer-only), 19.2 (with-steps).
• Providing step-by-step traces in the prompt improved cross-template accuracy by 3–15 points (average +6.0), highlighting the importance of explicit procedural supervision (Chen et al., 6 Jan 2026).
• In illustrative cases (e.g., the "distribute square-root over addition" malrule), large models predicted malrule-consistent responses for cross-template word problems at a rate of 40–60% using answer-only, rising to ~60% with step-trace prompting.

EDGE provides theoretical guarantees: if misconception clusters are constructed over joint item embeddings and a margin $\delta$ is enforced in counterfactual synthesis, then answering a contrastive item $q^\star$ correctly achieves an expected multiplicative reduction in the misconception posterior $\pi_{u,m}$ exceeding that for non-contrastive items by a factor $1+\kappa(\delta)$, uniformly over templates (Verma, 10 Aug 2025). This formalizes the leverage of cross-template items for misconception detection and remediation.

5. Cross-Template Dual-Path Trace Generation

A distinguishing methodological advance is the generation of dual-path traces for each problem instance:

• Correct trace ($S_c(i)$): step-by-step solution using valid mathematics
• Malrule-consistent trace ($S_m(i)$): step-by-step solution following the encoded student misconception

This enables both trace-level supervision and controlled analysis of where student reasoning diverges from the canonical procedure. For example, in fraction addition:

• Malrule “add numerators and denominators”:

$$\frac{1}{2} + \frac{1}{3} \to \frac{1+1}{2+3} = \frac{2}{5}$$

• Correct procedure:

$$\frac{1}{2}+\frac{1}{3} = \frac{3}{6}+\frac{2}{6} = \frac{5}{6}$$

Evaluation shows that supplying malrule-consistent step traces significantly boosts cross-template MRA, consistent with the hypothesis that procedural structure—rather than a single erroneous outcome—captures the essence of generalizable misconceptions (Chen et al., 6 Jan 2026).

6. Theoretical Guarantees and Adaptive Scheduling

EDGE introduces Bayesian Dirichlet mixture modeling of misconception classes and minimal-perturbation optimization for generating counterfactual items:

• Posterior inference aggregates evidence for latent misconception variables from error patterns on any template
• Theoretical analysis shows that counterfactual items with sufficient shortcut-contradicting margin offer provably superior posterior reduction, independent of template origin
• A restless-bandit index policy achieves near-optimal scheduling of practice and contrastive remediation across topics and templates, balancing mastery, retention, pace, confidence, and misconception eradication (EdgeScore is monotonic and Lipschitz-continuous in these components) (Verma, 10 Aug 2025)

This suggests a principled path for integrating cross-template misconception prediction into adaptive instructional sequencing.

7. Illustrative Example and Practical Implications

A concrete cross-template scenario is provided in MalruleLib:

• Malrule: distribute square-root over addition
• Source template: "Evaluate $f(x)=\sqrt{x^2+25}$ at $x=8$"—student answers $13$ via the malrule.
• Target template (word problem): "You walk 8 blocks east and 3 blocks north. What is the straight-line distance?"
  • Malrule-consistent prediction: $\sqrt{8^2+3^2} \to \sqrt{8^2}+\sqrt{3^2}=8+3=11$

Across large LLMs, the observed likelihood of making the correct cross-template prediction (i.e., the student applies the same buggy logic in the new context) is only ~40–60% for answer-only, but rises substantially with step-trace prompting.

In practice, both MalruleLib and EDGE enable carrying over misconception posteriors between templates, supporting immediate risk assessment for unseen item types and facilitating fine-grained, theoretically justified remediation strategies (Chen et al., 6 Jan 2026, Verma, 10 Aug 2025).

---

References:

• "MalruleLib: Large-Scale Executable Misconception Reasoning with Step Traces for Modeling Student Thinking in Mathematics" (Chen et al., 6 Jan 2026)
• "EDGE: A Theoretical Framework for Misconception-Aware Adaptive Learning" (Verma, 10 Aug 2025)