Evaluating Adaptive Personalization of Educational Readings with Simulated Learners

Abstract: We present a framework for evaluating adaptive personalization of educational reading materials with theory-grounded simulated learners. The system builds a learning-objective and knowledge-component ontology from open textbooks, curates it in a browser-based Ontology Atlas, labels textbook chunks with ontology entities, and generates aligned reading-assessment pairs. Simulated readers learn from passages through a Construction-Integration-inspired memory model with DIME-style reader factors, KREC-style misconception revision, and an open New Dale-Chall readability signal. Answers are produced by score-based option selection over the learner's explicit memory state, while BKT drives adaptation. Across three sampled subject ontologies and matched cohorts of 50 simulated learners per condition, adaptive reading significantly improved outcomes in computer science, yielded smaller positive but inconclusive gains in inorganic chemistry, and was neutral to slightly negative in general biology.

• The paper introduces a simulation framework that employs theory-grounded simulated learners to evaluate adaptive personalization in educational readings.
• It details an ontology-grounded content pipeline that maps learning objectives to fine-grained reading segments and adaptive assessments.
• Experimental results show domain-dependent impacts with significant gains in computer science and mixed effects in biology and chemistry.

Adaptive Personalization of Educational Readings: A Comprehensive Simulation Framework

Introduction

The paper "Evaluating Adaptive Personalization of Educational Readings with Simulated Learners" (2604.16744) introduces a framework to analyze and quantify the efficacy of adaptive personalization for educational readings by leveraging theory-grounded simulated learners. This research targets a gap in the evaluation of adaptive systems where reading material, rather than assessment or feedback items, is the primary instructional intervention. The simulation rigorously incorporates a blend of discourse comprehension, individual reader differences, misconception dynamics, and explicit knowledge tracking, thereby creating a closed-loop environment to assess adaptive instructional policies prior to live classroom deployment.

System Architecture and Methodology

Ontology-Grounded Content Pipeline

Central to the framework is the construction of a fine-grained ontology of learning objectives (LOs) and knowledge components (KCs) derived from open-access textbooks. The process involves LLM-augmented extraction and curation of LOs/KCs that structure all downstream content personalization, assessment, and mastery tracking. The Ontology Atlas, a browser-based interface, serves as both a visualization tool and an editing environment for ontology authoring, ensuring the alignment and granularity of course-specific ontologies across domains.

Adaptive Content Generation

After ontology creation, the textbook corpus is segmented into fine-grained and context-level reading chunks. These are labeled with LO/KC mappings through a combination of embedding-based retrieval and LLM adjudication, enabling precise alignment of reading material to individual knowledge domains. Adaptive readings and isomorphic assessments are programmatically generated to target one LO per cycle and avoid surface-form bias.

Theory-Grounded Simulated Learner Model

The simulated learner is engineered to exhibit crucial properties of real learners:

• Differentiated comprehension capabilities captured through DIME-style reader factors.
• Persistence and revision of misconceptions using Knowledge Revision Components (KReC).
• Reading comprehension grounded in the Construction–Integration (CI) framework, modeling proposition-level retention, integration, and distortion.
• An explicit epistemic boundary defines what a simulated learner can retrieve and apply during assessment, precluding LLM-style over-performance.

Two parallel learner states are maintained:

• A hidden state encapsulates true mastery, a vector of misconception weights, episodic traces, and reader/revision profiles.
• A tutor-visible Bayesian Knowledge Tracing (BKT) estimate driven exclusively by observed assessment responses.

Adaptation is achieved via BKT-driven personalization, with low mastery estimates triggering richer explanations, more examples, stronger misconception refutation, and finer readability matching. Control runs receive non-adaptive, default-configured reading material for comparison.

Experimental Design

Experiments were conducted in three distinct discipline ontologies: computer science, general biology, and inorganic chemistry. Each subject domain involved:

• Sampling of four LOs across two chapters.
• Cohorts of 50 matched simulated learners per condition (adaptive/control).
• Three iterative reading–assessment cycles per LO, each cycle comprising newly generated aligned multiple-choice questions.

Assessment and mastery outcomes were compared at the learner level, with adaptation policy as the sole independent variable due to isolated reading-driven state change.

Results

Subject-Level Outcome Analysis

The paper presents strong, statistically significant improvement in both observed quiz accuracy and BKT gain for computer science when using adaptive readings: accuracy increased from 0.834 (control) to 0.865 (adaptive) with $\Delta=+0.031$, 95% CI [$+0.004$, $+0.058$], $p=0.026$, $d_z=0.325$, and BKT gain rose from 0.416 to 0.454 with $\Delta=+0.038$, 95% CI [$+0.002$, $+0.074$], $p=0.037$, $d_z=0.304$. In inorganic chemistry, gains were positive but did not reach statistical significance ($+0.004$0 for accuracy, $+0.004$1). General biology showed nearly identical or slightly negative observed outcomes for adaptive relative to control, despite descriptive increases in underlying hidden mastery.

Figure 1: Subject-level adaptive-minus-control deltas in accuracy, BKT gain, hidden knowledge, and misconception reduction, with 95% confidence intervals.

Cycle-Level Trajectories

A detailed inspection of cycle-level results indicates that adaptive reading consistently leads in computer science (across all cycles) and in inorganic chemistry (especially during intermediate cycles), but does not maintain superiority in general biology, where control outpaces adaptive by the final cycle. This illustrates subject- and sequence-dependent effects of adaptation, further highlighting the criticality of content–ontology alignment and the stochasticity inherent to learner modeling.

Figure 2: Mean learner-level quiz accuracy by cycle for each subject and condition, with clear adaptive advantages in computer science and marginal/mixed effects in the other domains.

Discussion

The empirical evidence confirms that adaptive personalization, as instantiated in this framework, robustly improves learning outcomes in certain domains (notably computer science), but such benefits do not necessarily generalize universally. The simulator is sensitive to ontology granularity, LO/KC mapping, and domain-specific item bank properties, leading to domain-dependent effects which must be interrogated on a per-subject basis.

A key observation is the divergence between descriptive improvements in simulated hidden mastery and lack of significance in observed accuracy or BKT gain, particularly in biology. This underscores the nontrivial mapping between latent knowledge dynamics and observable assessment behavior—a function of both item calibration and the stochastic, multi-factor answer selection process (episodic retrieval, misconception weight, distractor competitiveness, etc.).

The framework, by disentangling observed and hidden metrics, provides a potent offline testbed for pre-deployment sensitivity analysis of adaptive instructional policies. It also demonstrates the value of a rich, editable ontology authoring tool (Ontology Atlas) for isolating sources of domain-specific variance and ensuring comprehensive coverage in adaptation evaluations.

Implications and Future Directions

The results have direct practical implications. Systems incorporating adaptive readings can expect measurable, though domain-dependent, benefits in learning outcomes—conditional on stringent ontology curation and content-alignment procedures. The methodological rigor of the simulator—especially the explicit epistemic boundary and fine-grained misconception dynamics—addresses concerns about the validity of LLM-driven student simulation and provides a foundation for safe, ethical policy screening before classroom integration.

Theoretically, the work suggests a bifurcation in future research: (1) iterated refinement of simulated learner fidelity, possibly by incorporating generative answer reasoning or fine-grained time/on-task/motivation parameters; and (2) the deployment of this simulation-in-the-loop paradigm for real-world adaptive system design, A/B testing, and high-throughput formative evaluation.

Conclusion

By operationalizing a theory-driven, transparent simulation pipeline for adaptive reading personalization, this paper advances both the evaluation methodology and the practical toolkit available for intelligent educational systems. The empirical findings substantiate statistically reliable benefits in computer science, domain-contingent efficacy in chemistry, and mixed or null effects in biology, with each outcome tightly bound to the nuances of knowledge representation and learner state modeling. This approach establishes an extensible, auditable paradigm for bridging NLP-based content generation with cognitively plausible personalization—laying groundwork for rational, data-driven policy decisions in AI-powered education.