IRT-Aligned Agents: Adaptive Evaluation

• IRT-aligned agents are computational models that apply item response theory to calibrate latent traits and item characteristics in adaptive evaluation settings.
• They combine methods such as joint embeddings, simulation via fine-tuned LLMs, and reinforcement learning for dynamic item selection to ensure robust parameter recovery.
• These models provide actionable insights in applications like educational assessments, LLM evaluations, and multi-agent benchmarks, enhancing efficiency and interpretability.

Item Response Theory (IRT)-Aligned Agents

Item Response Theory (IRT)-aligned agents are computational or neural architectures—often based on LLMs or related agentic systems—whose behaviors, calibration, or simulation mechanisms are explicitly parametrized or constrained to reflect the psychometric principles of IRT. These agents are designed for robust, interpretable measurement of latent traits (e.g., ability, competence) and item characteristics (e.g., difficulty, discrimination) in high-dimensional environments ranging from educational assessment to multi-agent evaluation or coding benchmarks. A diverse array of frameworks has emerged, ranging from geometric joint embeddings of models and items, direct simulation of IRT-calibrated response curves, amortised inference-guided adaptive agents, identity-link models for label-free comparisons, and DPO-aligned LLM simulators, to feature-augmented IRT in agentic settings.

1. Theoretical Foundations and Psychometric Background

IRT models the probability that agent $i$ correctly responds to item $j$ as a function of latent ability $\theta_i$, item difficulty $b_j$, and (optionally) other item parameters such as discrimination $a_j$ and guessing $c_j$. The canonical forms are:

• 1PL (Rasch): $P(y_{ij}=1) = \sigma(\theta_i - b_j)$
• 2PL: $P(y_{ij}=1) = \sigma(a_j(\theta_i - b_j))$
• 3PL: $$P(y_{ij}=1) = c_j + (1-c_j)\sigma(a_j(\theta_i - b_j))$$
• Nominal Response Model (NRM) for multiclass settings

Here, $\sigma(x) = 1/(1+e^{-x})$. IRT-aligned agents operationalize these constructs for both real and simulated entities, mapping model–item interactions onto probabilistic response curves governed by these parameters. In high-complexity domains, additional compositional structure and feature conditioning are incorporated, such as joint embeddings, agent–scaffold decomposition, or learned mappings from item features to difficulty parameters (Yao et al., 26 Sep 2025, Scarlatos et al., 7 Jul 2025, Ge et al., 1 Apr 2026, Ormerod, 5 Jan 2026, Robertson, 16 Oct 2025, Keurulainen et al., 2023).

2. Geometric and Embedding-Based Approaches

The JE-IRT framework exemplifies the geometric formalism, embedding both LLM agents and questions within a shared Euclidean space $j$0. Each question $j$1 has a vector $j$2 whose direction ($j$3) encodes semantic content and whose norm $j$4 encodes difficulty. Each agent $j$5 receives a learned ability vector $j$6. The probability of a correct response is given by:

$j$7

Difficulty is directly proportional to item norm; ability is multidimensional, corresponding to projections onto semantic directions. This geometry supports topical specialization, interpretable OOD prediction, and data-efficient onboarding of novel agents via fitting $j$8 with fixed item-side parameters. The learned joint space exhibits emergent clusters, partial alignment with human taxonomies, and reveals that a single scalar $j$9 is insufficient to explain performance diversity (Yao et al., 26 Sep 2025).

3. Simulation and Calibration via Fine-Tuned LLMs

IRT-aligned simulation is advanced by agents trained to reconstruct item characteristic curves (ICCs) through fine-tuning protocols such as Low-Rank Adaptation (LoRA). For example, in (Ormerod, 5 Jan 2026), Qwen-3 LLMs are fine-tuned to generate responses conditioned on discretized ability descriptors, producing synthetic ICCs across the ability spectrum. These curves, comprised of response probabilities at fixed intervals $\theta_i$0, are fitted to 2PL/NRM models:

$\theta_i$1

This parameter recovery is competitive with or exceeds classical field-testing on difficulty and discrimination, particularly for discrimination—a historically challenging parameter. Such simulation-based approaches enable IRT calibration in cold-start and low-data regimes across MCQ and, with greater complexity, open-ended domains (Ormerod, 5 Jan 2026, Scarlatos et al., 7 Jul 2025).

4. Preference Optimization and Simulated Respondents

In the SMART framework (Scarlatos et al., 7 Jul 2025), IRT-aligned agents are constructed by aligning simulated students to a fitted IRT model using Direct Preference Optimization (DPO). After fitting ground-truth IRT parameters to human data (GPCM for graded responses), the agent is fine-tuned on preference pairs where, for ability $\theta_i$2, real response $\theta_i$3 should be more likely than $\theta_i$4 if $\theta_i$5. The DPO loss pushes the simulated generator $\theta_i$6 to match the likelihood structure imposed by IRT. Once aligned, the simulator generates new responses that, when rescored and re-fit to an IRT model, yield accurate out-of-sample difficulty estimates for previously unseen items. This approach surpasses feature-based and SFT-based baselines on open-ended difficulty prediction (Scarlatos et al., 7 Jul 2025).

5. Adaptive and Amortised Design for IRT Agents

Adaptive agents leverage amortised inference and experiment selection to maximize efficiency in inferring abilities. In (Keurulainen et al., 2023), the agent employs:

• An amortised inference network $\theta_i$7 (outputs Gaussian mean/variance from response history)
• An experiment selection policy $\theta_i$8 trained via DRL (e.g., PPO) to choose the next item maximizing expected information gain (negative posterior entropy).

Training is performed on synthetic students, and at deployment, inference and selection are real-time feed-forward operations. Compared to classical OED, this agent achieves faster, more accurate ability estimation with reduced computational overhead and demonstrates clear information-efficiency gains: matching uncertainty levels with approximately half the number of administered items (Keurulainen et al., 2023).

6. Identity-Link and Label-Free IRT for LLM Evaluation

Evaluation settings without explicit ground truth employ identity-link IRT on raw scores derived from pairwise comparison statistics. The method in (Robertson, 16 Oct 2025) uses total variation distance mutual information (TVD-MI) from binary judge trials, yielding scores $\theta_i$9 per agent–item pair. The IRT model is:

$b_j$0

Fitting is performed using box-constrained least squares, enforcing $b_j$1 and a gauge constraint $b_j$2, with a small ridge penalty for stability. Empirically, the identity link preserves the additive structure of the data (median curl in $b_j$3), with substantially lower integrability violations than logistic or probit links. This approach achieves robust ability/difficulty calibration and model ranking (Spearman $b_j$4 for agent ability across sparse/dense settings) with significant evaluation cost reduction (Robertson, 16 Oct 2025).

7. Feature-Augmented and Decomposed IRT for Agentic Systems

Hybrid approaches in agentic coding, such as in (Ge et al., 1 Apr 2026), integrate IRT with feature regression and compositional agent modeling. Task difficulty parameters $b_j$5 are regressed from rich feature vectors $b_j$6 capturing semantic, rubric-based, and auditor-extracted properties. Agent abilities are decomposed into LLM and scaffold contributions: $b_j$7. Aggregating across heterogeneous leaderboards via this decomposition enables robust prediction for new (LLM, scaffold, task) combinations not previously observed jointly. The methodology supports adaptive evaluation via Fisher information maximization and delivers high AUC for ability ranking and OOD prediction (Ge et al., 1 Apr 2026).

Approach	Model Type	Core Innovation
JE-IRT (Yao et al., 26 Sep 2025)	Geometric IRT	Joint embeddings, multidimensional ability/difficulty
LoRA/ICC (Ormerod, 5 Jan 2026)	Simulation	LLM-simulated ICCs, synthetic response curves
SMART (Scarlatos et al., 7 Jul 2025)	DPO-aligned sim	IRT-likelihood DPO fine-tuning for student simulators
Amortised RL (Keurulainen et al., 2023)	DRL/VI	Amortised posterior, RL-driven item selection
Identity-link (Robertson, 16 Oct 2025)	Additive, label-free	TVD-MI, additive raw scores, Gini-entropy estimator
Agentic coding (Ge et al., 1 Apr 2026)	Feature-aug. IRT	Feature→difficulty regression, LLM+scaffold ability

8. Practical Implications and Application Scenarios

IRT-aligned agents provide rigorous, interpretable, and extensible evaluation and simulation capabilities for agentic systems, LLMs, and adaptive assessment environments. They enable model calibration (calibrated confidence, abstention, adaptive decoding), multi-agent routing (dynamic selection among specialized agents), pre-testing of novel benchmarks (cold-start difficulty estimation), and efficient adaptive evaluation (Fisher-optimal task selection). These instantiations have achieved state-of-the-art item parameter recovery, sample efficiency, and generalization across domains and modalities. Applications span educational assessment, LLM evaluation, coding benchmarks, curriculum design, and robust agent system monitoring (Yao et al., 26 Sep 2025, Ormerod, 5 Jan 2026, Scarlatos et al., 7 Jul 2025, Keurulainen et al., 2023, Robertson, 16 Oct 2025, Ge et al., 1 Apr 2026).

A salient implication is that multidimensional and feature-augmented IRT-aligned approaches enable finer-grained diagnostics than scalar ability estimates and unlock modular, data-efficient scaling to new agent classes and domains. However, the practical validity of these psychometric proxies, especially in high-stakes or OOD regimes, remains an active area for continued empirical verification.