Automated Item Generation (AIG) Overview

• Automated Item Generation (AIG) is a computational paradigm that systematically creates assessment items using explicit rules, templates, and modern language models.
• It integrates classical rule-based models with LLM-driven pipelines to ensure that items meet stringent psychometric and content validity criteria.
• AIG systems incorporate iterative validation, scoring architectures, and quality assurance metrics to deliver diverse, reliable, and secure item banks.

Automated Item Generation (AIG) is the use of computational procedures to systematically produce assessment items at scale while preserving alignment with construct definitions, content specifications, validity requirements, and, where applicable, psychometric targets. In classical psychometrics, AIG has usually been associated with rule-governed item models for constrained-response formats such as multiple-choice cognitive tasks; more recent work extends the concept to free-response creativity tasks, reading comprehension, personality situational judgment tests, medical certification vignettes, and even benchmark generation for evaluating AI systems (Jr. et al., 2024, Yang et al., 2022, Zheng et al., 21 Mar 2026). Across these settings, the central problem is not merely text generation, but the joint control of item structure, scoring, difficulty, diversity, and operational quality.

1. Foundations and conceptual scope

AIG in its most general sense is the algorithmic creation of test items from explicit representations of content, format, and task constraints. In educational assessment this may involve the end-to-end synthesis of a passage, stem, options, and scoring key; in psychometrics it may involve producing items that are valid, reliable, and traceable to a construct model rather than manually drafting each item de novo (Hwang et al., 19 May 2026, Jr. et al., 2024). In figural analogy research, AIG is described as the use of computer algorithms to systematically produce items at scale, with the explicit goal of controllable psychometric properties, secure item banks, and diversity of item families (Yang et al., 2022).

Historically, the dominant paradigm was model-based. Cognitive design systems, item shells, rule libraries, and parameterized templates defined which features could vary and how those variations should affect difficulty or construct representation. In figural analogy tasks, this tradition includes Hornke’s constrained rule-based construction, Embretson’s Cognitive Design System, MatrixDeveloper, GeomGen, the Sandia Matrix Generator, Wang’s CSP and first-order-logic formulation, and the IMak package (Yang et al., 2022). In medical certification, the same historical lineage appears in template-based and cognitive-model approaches for single-best-answer multiple-choice questions, before transformer-based generation was explored (Davier, 2019).

The contemporary literature broadens both the item formats and the generators. AIG now encompasses multiple-choice questions, cloze and short-answer items, constructed responses, free-response creative problem-solving scenarios, reading passages, and situational judgment tests (Hwang et al., 19 May 2026, Jr. et al., 2024, Bezirhan et al., 2023, Li et al., 2024). This expansion changes the technical problem. For constrained formats, correctness and uniqueness are often the key requirements; for open-ended formats, the system must also manage scoring models, content validity review, and robustness against evaluator bias. A plausible implication is that modern AIG is best understood as a family of generation-and-validation pipelines rather than a single generation technique.

2. Classical design paradigms: templates, rules, constraints, and grammars

The classical AIG literature is organized around explicit formal control. In figural analogy domains, items are generated by encoding analogical rules as transformations over attributes and structure, such as $A:B::C:D$ with $B=T(A)$ and $D=T(C)$, where $T$ may compose rotation, scaling, translation, and set operations (Yang et al., 2022). Rule classes include identity, progression, set or logical relations, distribution, and arithmetic, and they can be formalized at unary, binary, or ternary relational granularities (Yang et al., 2022). Difficulty is then manipulated through the number of rules, parameter magnitudes, perceptual organization, inter-rule interference, and distractor plausibility.

Several traditions emerged within this paradigm. Cognitive model-based generation links stimulus features to latent psychometric properties and iteratively calibrates them with human data; rule-based and parametric generation samples from operator libraries and parameter ranges; constraint-based search enforces validity and uniqueness through formal predicates; generative grammars separate scene structure from entities and layout; and programmatic composition directly implements rule pipelines in code (Yang et al., 2022). These approaches share an emphasis on explicit controllability and solver-based verification.

The same logic appears in later educational AIG systems, although with different surface features. MAFIG formalizes a reading-comprehension item as $x=(P,q,O)$, where $P$ is the passage, $q$ the stem, and $O=\{o_1,o_2,o_3,o_4\}$ the options under a multiple-choice format, and then imposes feature-level constraints on passage length, average sentence length, vocabulary CEFR band, factuality, reasoning complexity, and neutrality among options (Hwang et al., 19 May 2026). Hard adherence is summarized as

$J(x)=\sum_{i=1}^{m} w_i \cdot \mathbb{I}[c_i(x)],$

while item-level achievement is

$$\mathrm{AR}(x)=\frac{1}{m}\sum_{i=1}^{m}\mathbb{I}[c_i(x)] \times 100.$$

This retains the classical AIG principle that item generation should be governed by observable design variables rather than by uncontrolled surface fluency alone (Hwang et al., 19 May 2026).

A recurring advantage of these formal paradigms is solver-based validity. In figural analogies, automated solvers can check whether exactly one answer satisfies the intended rule set; in reading-comprehension AIG, feature-specific evaluators can determine whether each constraint is satisfied; in benchmark generation, exact, numeric, symbolic, or rubric-guided validators can separate core from non-core items (Yang et al., 2022, Hwang et al., 19 May 2026, Zheng et al., 21 Mar 2026). This continuity is important: even when generation becomes LLM-driven, high-quality AIG still depends on explicit verification layers.

3. LLMs and the extension of AIG to open-ended assessment

The major recent shift is the use of LLMs to generate semantically rich, context-sensitive items in domains where fixed templates are too restrictive. The Creative Psychometric Item Generator (CPIG) extends AIG into creativity assessment by generating free-response creative problem-solving scenarios, producing synthetic responses under multiple prompting styles, scoring originality with a roberta-based model trained to predict mean human ratings, and iteratively selecting high-performing items as few-shot exemplars for the next round (Jr. et al., 2024). Its selection step can be greedy, random, or constraint-based, with the latter balancing originality and semantic diversity via

$B=T(A)$0

and

$B=T(A)$1

This architecture generalizes AIG from “generate once from a template” to “generate, simulate responses, score, and optimize over iterations” (Jr. et al., 2024).

A related extension appears in personality situational judgment tests. GPT-4 was used to generate Chinese PSJT items aligned with Big Five facets, with prompt optimization and temperature tuning used to control the trade-off between creativity and coherence (Li et al., 2024). The best-performing configuration was Prompt v2 at temperature $B=T(A)$2, and the resulting 40-item instrument showed acceptable reliability for several facets, moderate-to-strong convergent validity with corresponding NEO-PI-R facets, and better discriminant validity than the comparison self-report measure (Li et al., 2024). The same paper shows that LLM-based AIG can operate beyond knowledge testing and into non-cognitive assessment, provided the prompts specify scenario realism, keyed options, and trait relevance.

In educational language assessment, LLMs are used both to generate items directly and to support human-centered quality control. A GPT-3 system generated fourth-grade reading passages conditioned on released PIRLS exemplars and audience cues such as “for a 10-year-old,” after which candidates were filtered by a Lexile-like text-difficulty analyzer and edited by human reviewers for grammatical and factual correctness (Bezirhan et al., 2023). In K–12 morphological assessment, a fine-tuned Gemma 2B model and an untuned GPT-3.5 baseline were evaluated under seven prompting strategies; structured prompting, especially chain-of-thought plus sequential design, improved construct alignment and pedagogical quality, with Gemma’s CoT + Sequential condition achieving the highest total pedagogical score among its prompt conditions (Amini et al., 27 Aug 2025).

The same trend is visible in medical certification. Fine-tuning GPT-2 on the PubMed Open Access subset was used to generate draft clinical vignettes and distractor proposals for multiple-choice questions (Davier, 2019). The outputs were not treated as deployable items; rather, they were used as SME-facing drafts because hallucinations, factual errors, and stylistic drift remained substantial. This contrast is instructive. LLMs are effective at producing domain-shaped text, but unrestricted generation does not by itself satisfy the requirements of operational AIG.

4. Psychometric control, validity evidence, and scoring architectures

AIG is fundamentally psychometric only when generation is coupled to evidence about validity, reliability, or operational functioning. In classical dichotomous-item settings, this often means IRT calibration. The figural analogy literature explicitly treats observable item features as predictors of latent item parameters, with 1PL, 2PL, and 3PL formulations used to link discrimination, difficulty, and guessing to generated stimulus features (Yang et al., 2022). A standard 2PL form is

$B=T(A)$3

This design philosophy underlies adaptive testing and targeted item-family generation.

Open-ended AIG often cannot rely on this machinery directly, so surrogate psychometric criteria are introduced. CPIG optimizes free-response CPS items using originality as the primary item-quality proxy: each generated item receives 10–20 synthetic responses, each response is scored by a roberta-base originality model trained to predict mean human originality ratings on a 5-point Likert scale, and the item score is the mean originality across responses (Jr. et al., 2024). Human content review then evaluates complexity and difficulty on 5-point scales, with inter-rater reliability reported as $B=T(A)$4 for complexity and $B=T(A)$5 for difficulty (Jr. et al., 2024). The system did not conduct IRT or Rasch calibration, and the paper explicitly notes that items were optimized through originality scoring and similarity constraints rather than latent-trait item models (Jr. et al., 2024).

In MAFIG, psychometric control is operationalized through feature adherence and empirical difficulty ordering rather than item-response calibration. The framework constructs a sequence of constraint sets intended to yield monotonically increasing difficulty and validates adjacent levels with a Difficulty Alignment Score: $B=T(A)$6 On automatic evaluation, MAFIG with Qwen3-32B achieved $B=T(A)$7, $B=T(A)$8, and $B=T(A)$9; in human evaluation it reached $D=T(C)$0 and $D=T(C)$1 (Hwang et al., 19 May 2026). These results show a distinct psychometric strategy: instead of estimating $D=T(C)$2 parameters from examinee response data, the generator enforces interpretable feature constraints and validates the induced ordering empirically.

Personality-SJT AIG uses yet another validation pattern. GPT-4-generated items were reviewed for content validity by psychology doctoral raters, including Content Validity Ratio,

$D=T(C)$3

with Prompt v2 reaching $D=T(C)$4, above Lawshe’s recommended $D=T(C)$5 threshold for $D=T(C)$6 experts (Li et al., 2024). Subsequent field testing used classical test theory and CFA rather than IRT. Reliability estimates included Cronbach’s alpha,

$D=T(C)$7

with values of $D=T(C)$8, $D=T(C)$9, $T$0, $T$1, and $T$2 across the five GPSJT facets, and CFA fit indices of $T$3, $T$4, $T$5, and $T$6 for the generated instrument (Li et al., 2024).

These examples show that AIG does not imply a single psychometric methodology. Depending on the domain, psychometric control may be achieved through latent-parameter calibration, feature-constrained design, automated scoring proxies, content-validity panels, factor-analytic evidence, or combinations of these. The common requirement is that generated items must be evaluated against explicit measurement criteria rather than only surface plausibility.

5. Operational pipelines, quality assurance, and item-bank management

Operational AIG is not exhausted by generation. It includes screening, human review, piloting, calibration, deployment, and post-deployment monitoring. AQuAP presents this explicitly as the analytics layer coupled to Duolingo’s Item Factory, a human-in-the-loop AIG framework in which SMEs specify task families and blueprints, AI models generate content, automated screening checks length, grammar, spelling, format conformance, and basic validity, and human review performs Item Quality Review and Fairness and Bias review before piloting and operational integration (Davier et al., 16 Jun 2026).

AQuAP formalizes item-bank health with structural and usage-sensitive metrics. Effective Bank Size is defined as

$T$7

and aggregate EBS is a time-weighted combination across item types (Davier et al., 16 Jun 2026). Because adaptive delivery produces uneven exposure, Adjusted Effective Bank Size multiplies EBS by an entropy-based Effective Bank Use term: $T$8 Additional metrics include Maximum Exposure, Maximum Conditional Exposure, and Rarely-Administered Fraction (Davier et al., 16 Jun 2026). This operational layer closes the loop between generation and delivery by determining where new content is needed, which items are overexposed, and where the bank is structurally weak.

The same separation between generation and governance appears in practice-test systems. In the Duolingo English Test, practice tests are generated with modern GPT-based AIG methods, pass automated linguistic and social-appropriateness checks, receive human copyediting, fact-checking, and fairness review, and are then assembled adaptively from a pool separate from the operational test bank (Burstein et al., 23 Aug 2025). In a large observational sample of $T$9, taking $x=(P,q,O)$0 to $x=(P,q,O)$1 practice tests was associated with higher official DET scores and more positive affect, while taking more than $x=(P,q,O)$2 was associated with lower performance, which the paper interprets as potential washback (Burstein et al., 23 Aug 2025). The direct AIG implication is that large generated pools enable repeated practice, but operational policy must still manage security, usage patterns, and construct-irrelevant variance.

Benchmark-generation research generalizes the same logic to AI evaluation. BenchBench treats automated benchmark generation as a three-stage AIG pipeline: extract domain cards from seed benchmarks, generate quota-controlled suites with multiple designer models, and validate items with a multi-model answerer panel using objective-first scoring and quality gates (Zheng et al., 21 Mar 2026). Across 16,669 generated items and 14,893 retained core items, invalidity was negatively associated with discrimination, with Pearson $x=(P,q,O)$3, and design ability correlated only moderately with answer-time strength, with Spearman $x=(P,q,O)$4 (Zheng et al., 21 Mar 2026). This suggests that operational quality assurance is not an auxiliary concern; it is constitutive of whether generated items are diagnostically useful at all.

6. Limitations, controversies, and future directions

A persistent limitation is that fluency is easier to automate than validity. In medical certification, transformer-generated vignettes were clinically styled but contained hallucinations, inappropriate therapies, inconsistent lab values, and other factual problems, so SME review remained mandatory (Davier, 2019). In reading passage generation, factual and numerical claims required editorial checking even after readability matching (Bezirhan et al., 2023). In creativity assessment, the originality scorer used by CPIG had Pearson $x=(P,q,O)$5 with human ratings on held-out items and was trained on data reflecting a Western cultural perspective, which the paper identifies as a possible source of evaluator bias (Jr. et al., 2024). In MAFIG, option-level neutrality and reasoning-complexity constraints were the hardest to satisfy, and evaluator noise remained a major sensitivity point (Hwang et al., 19 May 2026).

A second controversy concerns shortcut validity. Synthetic benchmarks and algorithmically generated distractors can inadvertently create answer-set regularities that allow solving without engaging the intended task. In the RAVEN literature, answer-set shortcuts enabled context-blind models to exceed 70–90% accuracy on the original formulation, motivating Impartial-RAVEN and RAVEN-FAIR (Yang et al., 2022). BenchBench addresses a related issue by preferring exact, numeric, and symbolic verification over unconstrained LLM judging and by removing items flagged as not well posed, gold incorrect, or ambiguous (Zheng et al., 21 Mar 2026). These results imply that AIG systems must audit not only content quality but also exploitability.

Fairness and cultural validity remain open problems. Personality-SJT generation in Chinese did not report DIF analyses or subgroup invariance testing (Li et al., 2024). DET practice-test research reported demographic comparability to the wider DET population but did not conduct subgroup-specific outcome analyses or measurement-invariance testing (Burstein et al., 23 Aug 2025). AQuAP emphasizes fairness and bias review during item development and recommends integrating exposure metrics with DIF and fairness dashboards (Davier et al., 16 Jun 2026). This suggests that scaling generation without scaling fairness auditing produces an incomplete AIG pipeline.

The dominant future directions are explicit in the recent literature. CPIG points toward multi-criteria optimization over originality, effectiveness, and feasibility, together with multi-evaluator ensembles and cross-cultural scoring datasets (Jr. et al., 2024). MAFIG points toward stronger evaluator ensembles, curriculum-style constraint scheduling, and eventual linkage of calibrated constraint levels to empirical item difficulty parameters (Hwang et al., 19 May 2026). Figural analogy research continues to motivate hybrid systems that combine symbolic generators, formal solvers, and psychometric calibration (Yang et al., 2022). BenchBench proposes expanded formal verification, multi-judge calibration, broader domains, and eventual human-cohort calibration under models such as 2PL or 3PL (Zheng et al., 21 Mar 2026). Across domains, the trajectory is toward closed-loop AIG systems in which generation, validation, psychometric analysis, fairness review, and bank-health monitoring are treated as one integrated measurement process rather than as separate engineering tasks.