Validate DIF methodology for small, dependent LLM groups

Establish the adequacy of group-specific item estimation, anchor selection, and uncertainty calibration for differential item functioning analyses involving small, dependent groups of large language models, including determining whether simpler Mantel–Haenszel and ridge-logistic screens remain reliable in this setting.

Background

The paper compares OpenAI and Anthropic models using differential item functioning analyses after matching models on estimated refusal ability. Because the comparison includes only 21 OpenAI models and 11 Anthropic models, and because models from the same developer share training pipelines and are not independent examinees, standard procedures for group-specific item estimation, anchor selection, and uncertainty calibration may not be adequate.

The authors therefore retain Mantel–Haenszel and ridge-penalized logistic screens as simpler conditional methods rather than extending the analysis to multiple-group item response theory with anchor purification. They explicitly identify the methodological adequacy of these procedures as unresolved for small, dependent LLM groups, noting that the simpler screens do not avoid the underlying issues.