Capability-Aligned Rater Decomposition

• The paper introduces a unified model that decomposes rater performance into capability-specific components to capture sensitivity to latent abilities.
• It details both scalar and functional profiling using logistic models and Laplace approximation for efficient, parameter estimation in diverse contexts.
• The methodology extends to multimodal data curation, employing meta-learned raters to improve model training and selection in large-scale assessments.

Capability-Aligned Rater Decomposition encompasses a set of methodological frameworks and techniques for modeling, quantifying, and operationalizing rater judgment sensitivity with respect to targeted competencies or latent traits. This approach generalizes traditional rater-indexing and monolithic data scoring by decomposing rater performance into capability-specific components, enabling both scalar and functional assessment of alignment between rater signal and the underlying abilities or capabilities of subjects, examinees, or data samples. The principle of capability-alignment, as formalized in both psychometric assessment models (Wang et al., 13 Feb 2025) and multimodal data curation (Sahi et al., 12 Feb 2026), underpins rigorous analysis, optimization, and interpretation of rater or scoring system effectiveness along distinct and potentially orthogonal skill axes.

1. Theoretical Foundations and Differential Capability Index

In educational assessment, the Capability-Aligned Rater Decomposition process is anchored in the generalized multi-facet (GMF) framework, which integrates rater discrimination, severity, and item difficulty into a parametric logistic formulation. For rater $r$ and item $i$, and ability $\theta$:

$$P_{ri}(\theta) = \frac{\exp(\rho_r \theta - \delta_i - \eta_r)}{1 + \exp(\rho_r \theta - \delta_i - \eta_r)}$$

with $\rho_r$ (discrimination), $\delta_i$ (item difficulty), and $\eta_r$ (severity).

The principal capability measure is defined via the point-wise derivative of passing probability with respect to ability:

$$C_i(\theta) = \frac{\partial P_i(\theta)}{\partial\theta} = \alpha_i P_i(\theta)[1 - P_i(\theta)]$$

where $\alpha_i = \rho_r$ and $b_i = (\delta_i + \eta_r)/\rho_r$.

In the hierarchical rater model (HRM) extension, the capability index reflects the sensitivity of observed rater outcomes to changes in latent subject ability through both rater and latent-response pathways. The general differential form is:

$$\frac{\partial P(Y=1)}{\partial\theta} = [F(c_r - a_{ri}) - F(c_r)] F_1(1 - F_1)$$

where $F_1 = \mathrm{logistic}(\theta - \delta_i)$.

This construction allows the quantification of rater sensitivity as a function of examinee ability, resolving deficiencies in indices that aggregate severity or discrimination in isolation (Wang et al., 13 Feb 2025).

2. Functional and Scalar Decomposition Across Contexts

The methodology enables both scalar and functional capability profiling. For each rater-item pair, the capability curve $C_i(\theta)$ is computed on a dense ability grid, assembling a capability profile $\kappa_r(\theta)$ for each rater. Summarization includes:

• Functional Comparison: $\kappa_{r, \text{topic}}(\theta)$ curves are overlaid by topic/context or decomposed via principal components, enabling fine-grained comparison of rater sensitivity as a function of ability.
• Scalar Integral Summary: The overall capability index $$\bar\kappa_r = \int \kappa_r(\theta)\phi(\theta)\,d\theta$$ (using standard normal weight) enables rater ranking and cross-context comparison.

Variation in rater severity ($\eta_r$) shifts the peak of the capability curve, while discrimination ($\rho_r$) alters its steepness and peak height. Under the total-facet model (TFM, $\rho_r\equiv1$), only severity-tailored shifts are present; under GMF, the full spectrum of sensitivity and alignment is represented. Simulation demonstrates $\bar\kappa_r$ maximizes when severity is centered ($\eta_r \approx 0$) and declines otherwise (Wang et al., 13 Feb 2025).

3. Computational Implementation: Estimation and Scaling

Efficient parameter estimation is achieved by marginal likelihood maximization, employing Laplace approximation around the mode $\theta_n^*$. This allows for tractable processing even for large or incomplete $N\times R$ data matrices via matrixized and parallelized operations.

Key computational features include:

• Marginal log-likelihood evaluation and Laplace-approximated integrals:

$$LL_{LAP} \approx \sum_n \left\{ h(\theta_n^*) - \frac{1}{2}\log|h''(\theta_n^*)| \right\}$$

with $h(\theta_n)$ as the log-posterior.

• Parameter recovery validation in simulation, with bias and RMSE quantified over 200 replications for $(\rho_r, \eta_r, \theta_n)$.
• Applicability to incomplete designs using sparse data structures and parallel computation (Wang et al., 13 Feb 2025).

4. Extensions to Multidimensional Data: Multimodal SkillRater

SkillRater (Sahi et al., 12 Feb 2026) generalizes capability-aligned rater decomposition to multimodal data curation by constructing one rater $r_{\phi_c}$ per capability $c$, each implemented as a parameter- and compute-efficient transformer atop frozen encoder features. This approach repurposes the capability-aligned decomposition to the model-training data selection pipeline, where each rater is meta-learned via bilevel optimization:

$$\min_{\phi_c}\;\mathcal{L}_{\mathrm{val}^{(c)}}(\theta^{(S)}(\phi_c))$$

subject to

$$\theta^{(S)}(\phi_c) = \text{InnerLoop}(\theta^{(0)}, r_{\phi_c}, \mathcal{D}_{\mathrm{train}})$$

with $r_{\phi_c}(z)$ scoring each sample for capability $c$.

During data curation, samples are retained if any rater scores them above a dynamically decaying threshold, as per the progressive union-curriculum:

$$\mathrm{retain}(z, t) = \bigvee_{c=1}^{C} \left[ r_{\phi_c}(z) \geq \tau_c(t) \right]$$

where $\tau_c(t)$ is set such that the top $p(t)$ fraction passes per rater, and $p(t)$ follows a quadratic decay schedule.

Empirical results confirm that rater outputs are nearly orthogonal, as measured by low inter-rater Pearson and Spearman correlations and PCA effective dimensionality (2.99/3 for three capabilities). This substantiates the interpretability and independence of capability axes captured via rater decomposition (Sahi et al., 12 Feb 2026).

5. Empirical Application and Outcomes

Educational assessment application (Wang et al., 13 Feb 2025) involves essays from 363 students on four topics, rated by four raters. The GMF fit yields per-rater $(\rho_r, \eta_r)$, and $\bar\kappa_{r, \mathrm{topic}}$ allows comparison:

Topic	$\bar\kappa_{r, \mathrm{topic}}$
Family	0.76
School	0.70
Work	0.68
Sport	0.64

Raters exhibit highest alignment on "family" and lowest on "sport", with individual differences substantial by rater-topic pair. This functional analysis affords insights into content-domain effects and rater specialization, unattainable via severity or discrimination alone.

In SkillRater's multimodal context (Sahi et al., 12 Feb 2026), capability decomposition leads to substantial performance improvements over monolithic filtering. Held-out mean accuracy measured gains: +5.63% (visual), +2.00% (OCR), +3.53% (STEM) at 2B-parameter scale. Curriculum-based filtering further outperforms all static top-k schemes.

6. Synthesis and Interpretive Implications

Capability-Aligned Rater Decomposition offers a principled foundation for unifying traditional rater metrics into a single, ability-aware framework. Key advances include:

• Provision of a closed-form, interpretable index $C_i(\theta)$ capturing local rater sensitivity to ability.
• Construction of rater capability profiles as functional objects, supporting intra- and inter-contextual comparison.
• Generalization to multidimensional capability axes in large-scale data regimes via meta-learned, orthogonal raters.
• Algorithmic efficiency via Laplace approximation and matrixized computation for scalability.
• Empirical gains in both psychometric instrument evaluation and model-training data curation.

A plausible implication is that this decomposition paradigm establishes a systematic substrate for analyzing, enhancing, and designing both human and automated rating systems wherever multidimensionality and contextual alignment are pivotal (Wang et al., 13 Feb 2025, Sahi et al., 12 Feb 2026).