Assessing and Mitigating Miscalibration in LLM-Based Social Science Measurement

Abstract: LLMs are increasingly used in social science as scalable measurement tools for converting unstructured text into variables that can enter standard empirical designs. Measurement validity demands more than high average accuracy, which requires well calibrated confidence that faithfully reflects the empirical probability of each measurement being correct. This paper studies the model miscalibration in LLM-based social science measurement. We begin with a case study on FOMC and show that confidence based filtering can change downstream regression estimates when LLM confidence is miscalibrated. We then audit calibration across 14 social science constructs covering both proprietary models, including GPT-5-mini, DeepSeek-V3.2, and open source models. Across tasks and model families, reported confidence is poorly aligned with tolerance-based correctness. As a simple mitigation, we propose a soft label distillation pipeline for calibrating Bert with LLM. The method converts an LLM score and its verbalized confidence into a soft target distribution, then trains a smaller discriminative classifier on encoder models for these targets. Averaged across datasets, this approach reduces ECE by 43.2\% and Brier by 34.0\%. These results suggest that LLM-based social science pipelines should treat calibration as part of measurement validity, rather than as an optional post-processing concern.

• The paper demonstrates that miscalibrated LLM confidence leads to significant regression coefficient attenuation, undermining measurement validity.
• It systematically evaluates calibration across diverse LLMs using metrics like T-ECE, Brier Score, and rank-order correlation.
• The study introduces a soft label distillation approach that reduces ECE by 43.2% and Brier Score by 34.0%, enhancing measurement reliability.

Calibration of LLM-Based Social Science Measurement: Systematic Assessment and Distillation

Motivation and Problem Framing

The use of LLMs for automated conversion of unstructured text into empirical constructs is ubiquitous in social science. Measurement validity in these applications not only hinges on strong predictive alignment with human labels but also on proper calibration—i.e., the concordance between reported confidence and the empirical probability of correctness. Miscalibrated LLM confidence can compromise downstream statistical inferences, especially when confidence-thresholding is employed for data selection prior to regression or policy analysis. This work interrogates calibration as an underappreciated reliability bottleneck in LLM-based measurement pipelines, systematically diagnosing miscalibration and proposing practical mitigation strategies.

Empirical Evidence: Miscalibration Distorts Regression Estimation

A case study using FOMC (Federal Open Market Committee) policy stance measurement demonstrates the downstream impact of model miscalibration. LLM-generated stance scores filtered by high confidence yield systematically attenuated regression coefficients relative to ground truth, reducing effect sizes and explaining less variance (R²). Specifically, filtering by confidence > 90 reduces the OLS coefficient for CPI-driven policy stance from 0.0855 (human-labeled ground truth) to 0.0261, with corresponding drops in fit ($R^2$ from 0.7012 to 0.3963). The high t-statistics indicate the relationship persists, but sample selection based on miscalibrated confidence induces attenuation bias—underscoring calibration as a central concern for empirical validity in computational social science.

Definitions and Metrics

Calibration is formalized using tolerance-based criteria. For measurement $y$ and confidence $q$, perfect calibration requires $$P(|y_\text{pred} - y_\text{true}| < \epsilon \mid \text{conf} = q) = q$$ for all $q$. Empirical diagnostics use (i) Tolerance-based Expected Calibration Error (T-ECE), (ii) Brier Score, and (iii) Machine Human Correlation (MH/Spearman) to separately quantify alignment, absolute calibration, and predictive rank-ordering.

Audit Across Models, Constructs, and Confidence Proxies

A comprehensive calibration audit is performed across 14 constructs spanning offensiveness, stance, sentiment, emotion, argument quality, and other text attributes. Eight LLMs—both API and open source—are evaluated using verbalized confidence, resampling, and logit-based proxies. Results reveal pervasive miscalibration regardless of inference paradigm:

• Verbal confidence elicited via prompt augmentation consistently deviates from ideal calibration.
• Logit-based proxies (geometric mean, p-true) sometimes reduce T-ECE but do not resolve absolute calibration gaps.
• Predictive alignment (MH/Spearman) is not predictive of calibration; top-performing models on rank correlation often show severe miscalibration.

The distributional mismatch and non-linear scaling are visually confirmed in reliability diagrams.

Post-hoc Calibration: Resolution Collapse

Standard post-hoc recalibrators—Platt scaling, Beta calibration, Isotonic regression, Temperature scaling—are benchmarked. While parametric scaling (Platt/Beta) drastically reduces T-ECE, this is frequently achieved at the cost of collapsing confidence resolution to a dataset base rate, thus destroying discriminative information. Brier Score confirms that such recalibrators optimize ECE but provide uninformative estimates. Temperature scaling is comparatively superior but still limited in maintaining confidence distribution integrity.

Soft Label Distillation for Calibration Mitigation

To address miscalibration, the authors propose a soft label distillation pipeline: LLMs produce measurement scores and self-verbalized confidences, which are transformed into soft target distributions for BERT-based classifier training. This approach regularizes confidence signals and produces more stable calibrated proxies. Across all datasets, ECE is reduced by 43.2% (from 0.408 to 0.228), and Brier Score by 34.0% (from 0.376 to 0.248). Soft label distillation mitigates prompt-level noise and facilitates probabilistic filtering.

Implications and Limitations

The systematic calibration gap identified in LLM-based measurement threatens empirical rigor, indicating that high-confidence filtering should not be naively adopted. Calibration should be foregrounded as a validity requirement in social science workflows, and both ECE and Brier Score ought to be reported alongside task performance. The findings have practical significance for automated policy analysis, behavioral measurement, and content moderation. Limitations include restriction to English text modalities and uninvestigated alignment techniques for uncertainty distribution preservation. Future directions include scaling into multilingual/multimodal domains and intrinsic calibration-aware model training.

Broader Impacts

The work improves reliability and transparency in computational social science, reducing bias in confidence-driven selection. Risks include misuse in sensitive applications where automated scoring could amplify latent biases or lend unjustified certainty. Mitigation necessitates caution, especially in politically charged or toxic content domains.

Conclusion

This study establishes calibration as a critical bottleneck for LLM-based social science measurement, demonstrating that predictive alignment is insufficient: confidence miscalibration directly distorts empirical regression estimates, threatens policy evaluation, and undermines measurement validity. Soft label distillation offers a practical means to mitigate calibration deficits, enhancing reliability for downstream filtering and inference. These results motivate broader reporting of calibration metrics and caution against uncritical reliance on LLM confidence for empirical workflow design.