Locating acts of mechanistic reasoning in student team conversations with mechanistic machine learning

Abstract: STEM education researchers are often interested in identifying moments of students' mechanistic reasoning for deeper analysis, but have limited capacity to search through many team conversation transcripts to find segments with a high concentration of such reasoning. We offer a solution in the form of an interpretable machine learning model that outputs time-varying probabilities that individual students are engaging in acts of mechanistic reasoning, leveraging evidence from their own utterances as well as contributions from the rest of the group. Using the toolkit of intentionally-designed probabilistic models, we introduce a specific inductive bias that steers the probabilistic dynamics toward desired, domain-aligned behavior. Experiments compare trained models with and without the inductive bias components, investigating whether their presence improves the desired model behavior on transcripts involving never-before-seen students and a novel discussion context. Our results show that the inductive bias improves generalization -- supporting the claim that interpretability is built into the model for this task rather than imposed post hoc. We conclude with practical recommendations for STEM education researchers seeking to adopt the tool and for ML researchers aiming to extend the model's design. Overall, we hope this work encourages the development of mechanistically interpretable models that are understandable and controllable for both end users and model designers in STEM education research.

• The paper presents a novel hierarchical switching-state recurrent dynamical model (HSRDM) that integrates classifier feedback to detect mechanistic reasoning in team discussions.
• It employs Markovian dynamics and variational Bayesian inference to capture time-varying, entity-level MR probabilities in collaborative STEM environments.
• Ablation studies demonstrate substantial performance improvements, underscoring the importance of explicit inductive bias for robust, interpretable MR detection.

Interpretable ML for Locating Mechanistic Reasoning in Student Team Conversations

Problem Motivation and Context

STEM education research increasingly prioritizes scalable, nuanced, and interpretable analysis of students’ in situ mechanistic reasoning (MR) during collaborative problem solving. Manual annotation of conversations for mechanistic reasoning evidence is labor-intensive and does not scale within domains like engineering education, where large corpora of student discussions are often available. Most existing ML approaches—including recent LLM-powered tools—exhibit weak interpretability guarantees and fail to capture time-evolving, entity-specific MR dynamics in multi-agent environments. The paper "Locating acts of mechanistic reasoning in student team conversations with mechanistic machine learning" (2604.21870) systematically addresses these gaps by developing a task-aligned, inherently interpretable hierarchical switching-state recurrent dynamical model (HSRDM) that outputs per-student, time-varying probabilities of mechanistic reasoning.

Modeling Framework and Methodology

The proposed approach is anchored in the HSRDM framework [wojnowicz2024discovering], which posits latent discrete state variables at both the group (system-level, with $L=2$ states) and individual (entity-level, with $K=4$ states) levels. Each student in a team is modeled as an entity whose observed utterances at each timestep are embedded with the EmbeddingGemma encoder (128 dimensions; parameter-efficient Matryoshka architecture). Transitions in latent state are governed by Markovian dynamics with conditional independence between entities given the group state, and autoregressive Gaussian emission over embedding vectors.

A critical distinctive component is the explicit, supervised inductive bias mechanism: a separate classifier, trained on subset-labeled utterances, predicts the maximum MR evidence category (via Russ’ seven-level taxonomy) for each utterance. This classifier’s output probabilistically modulates both group- and entity-level state transitions at every timestep. The feedback functions $g$ and $f$, together with largely fixed (non-learned) coefficient matrices, steer the HSRDM toward domain-aligned update rules—ensuring, e.g., that MR-by-one-student increases subsequent MR probabilities for self and peers, and that silent states are handled deterministically.

Figure 1: A toy illustration of the desired HSRDM probabilistic dynamics; MR evidence in one student’s utterance triggers sharp increases in that student’s and modest increases in their teammates' MR probabilities at the next timestep, compared to baseline drift.

All model parameters are learned with variational Bayesian inference, and the model is robustly initialized with emission parameters estimated via human-annotated MR labels (as opposed to random k-means centroids). The entire training procedure is deterministic.

Evaluation and Results

Evaluative focus is on metrics that tightly match the inductive bias and the underlying theoretical desiderata:

• H(i): Correlation between MR in an utterance and the (next-step) probability that the same student is in the MR-labeled silent state ($S1$).
• H(ii): Difference (“gap”) in $S1$ probability for speakers who just uttered MR evidence versus those who did not.
• H(iii): Gap in MR state ($T1$) probability for non-speakers who, at the previous step, heard an MR utterance, versus those who did not.

Ablation studies compare (A) model with classifier feedback, (B) model with only entity-level feedback (system-level removed), (C) model without feedback, and (D) model using oracle human feedback. Both seen and novel student teams and problems are used in evaluation.

Figure 2: Graphical structure of the HSRDM, showing hierarchical Markovian dependencies, conditional independence, and integrated classifier-based feedback mechanisms driving both group and entity state transitions.

Strong Empirical Findings

• Incorporation of classifier-derived feedback yields a 2x (seen) and 1.5x (unseen problem) increase in H(i) correlation metrics compared to zero-feedback variants. The mean $S1$ probability gap (H(ii)) is several times larger with the full model—up to $86\times$ (seen) and $313\times$ (unseen problem)—than with weaker or no inductive bias.
• Negative correlations are observed for non-MR-aligned states, confirming that the model allocates probability as theoretically intended.
• Removing hierarchical group structure (system state) degrades performance, confirming the utility of hierarchical modeling for capturing coordination.

Edge cases—e.g., small group sizes in test data, or test segments with very little MR—show degraded (or even paradoxically reversed) performance for H(iii); future work should address data scarcity for robust evaluation of these finer-grained inter-entity influence mechanisms.

Figure 3: Output of model with classifier feedback, showing alignment between human annotations of MR evidence, classifier predictions, and posterior per-student MR probabilities over a high-MR-density segment of conversation.

Numerics, Calibration, and Interpretability

• All classifier and model variants are thoroughly evaluated on accuracy and calibration for each of the 8 Russ code classes.
• The variational objective (ELBO) is shown to consistently increase and stabilize over training; convergence of the correlation metric between MR annotations and $K=4$0 probability is attained early (15 iterations).
• Codebase and pre-trained weights for all model variants are published, with user guidance and embeddings made available subject to IRB constraints.

Figure 4: Negative ELBO training curve demonstrating stable optimization and absence of overfitting in the HSRDM.

Figure 5: ELBO progress over 15 CAVI (coordinate ascent variational inference) iterations illustrating expected monotonic improvement and convergence.

Implications for Theory and Practice

The integration of strong, explicitly encoded inductive bias in a probabilistic entity-process model enables calibrated, interpretable, and domain-aligned tracking of MR in multi-party STEM conversational data. The model supports post hoc analysis (e.g., surfacing high-MR transcript regions for human review) and direct ingest into education research workflows, alleviating the bottleneck of manual MR annotation.

Practically, the methodology bridges the interpretability gap of black-box LLM approaches for time-series entity-level tasks, and provides robust guarantees about how evidence flows through the model. The ablation studies confirm that without inductive bias, modeling trends revert to non-informative behaviors, underscoring that interpretability and generalization here are consequences of explicit model design rather than fragile post-processing.

On the theoretical side, this framework provides a platform for the inclusion of richer priors, context-dependent annotation schemes, or multi-modal data streams (e.g., gesture, sketch), supporting future research into entity interaction modeling and domain adaptation.

Limitations and Forward Paths

Limitations include data scarcity (especially for larger groups), annotation cost (inductive bias still ultimately traces to human coding), and sensitivity to changes in task/problem context. Future work should address:

• Scalability to larger or more heterogeneous group sizes
• Incorporation of richer (e.g., contextual, multi-task) MR codes and modalities
• Systematic comparative benchmarking against LLM-ICL and alternative interpretable models
• Exploration of personalized entity parameters and non-Gaussian emission distributions
• Refinement of silent-state embeddings and inclusion of empirically motivated priors

The possibility to tightly integrate human-in-the-loop annotation for active model improvement or to expand to additional discourse phenomena (confusion, analogy, metacognition) is significant.

Conclusion

This work provides a rigorous template for combining structured, domain-aligned probabilistic models with explicit, supervised inductive bias—demonstrating strong interpretability, quantitative validation, and practical utility for automatic detection and analysis of mechanistic reasoning in collaborative STEM conversation. The accompanying code and best practices documentation position the model as an accessible research tool for both education and ML researchers.

The model’s hierarchically controlled, feedback-modulated temporal inference offers a principled and extensible alternative to black-box methods for educational time series data, exemplifying a broader shift toward inherently interpretable, mechanism-centric ML in high-stakes domains.