Measurement Risk in Supervised Financial NLP: Rubric and Metric Sensitivity on JF-ICR

Abstract: As LLMs become credible readers of earnings calls, investor-relations Q&A, guidance, and disclosure language, supervised financial NLP benchmarks increasingly function as decision evidence for model selection and deployment. A hidden assumption is that gold labels make such evidence objective. This assumption breaks down when the benchmark ruler itself is sensitive to rubric wording, metric choice, or aggregation policy. We study this measurement risk on Japanese Financial Implicit-Commitment Recognition (JF-ICR; a pinned 253-item test split x 4 frontier LLMs x 5 rubrics x 3 temperatures x 5 ordinal metrics). Three findings follow. First, rubric wording materially changes model-assigned labels: R2--R3 agreement ranges from 70.0% to 83.4%, with the dominant movement near the +1 / 0 implicit-commitment boundary. This pattern is consistent with a pragmatic-boundary interpretation, but is not a validated linguistic-causality claim because the present rubric variants confound semantics, examples, and verbosity. Second, not every metric remains informative under the JF-ICR class distribution. Within-one accuracy is too easy because near misses receive credit and the majority class dominates; worst-class accuracy is too noisy because the rarest class has only two examples. Exact accuracy, macro-F1, and weighted \k{appa} are therefore the identifiable metrics under our operational rule. Third, ranking claims become more defensible only after this metric-identifiability audit: Bradley--Terry, Borda, and Ranked Pairs agree on the identifiable metric subset, while the full five-metric sweep produces disagreement on the closest pair. The contribution is not a new leaderboard, but a reporting discipline for supervised financial benchmarks whose gold labels exist and whose evaluation ruler still requires governance.

• The paper demonstrates that rubric variation significantly influences financial NLP benchmark outcomes by inducing label shifts of up to 27.5%.
• It establishes a deterministic metric identifiability checklist to ensure robust ranking amidst class imbalance and measurement risk.
• The study advocates rigorous audit protocols for transparent operational reporting in financial NLP system deployment.

Measurement Risk in Supervised Financial NLP: Rubric and Metric Sensitivity on JF-ICR

Motivation and Framing

The paper addresses the critical issue of measurement risk in supervised financial NLP, specifically within the context of model evaluation and selection for tasks grounded in Japanese financial disclosure language. As LLMs gain traction among financial institutions for ingesting earnings calls, IR Q&A, and guidance statements, benchmark results are increasingly used as actionable evidence in procurement and deployment decisions, extending far beyond research diagnostics.

The core argument is that the assumption of benchmark objectivity, grounded in gold labels, is fragile. The evaluation “ruler”—comprising rubric definition, aggregation metric, and ranking policy—can yield substantially different outcomes depending on its construction. The authors focus on Japanese Financial Implicit-Commitment Recognition (JF-ICR), a 5-class ordinal classification problem, and propose an audit protocol to disentangle rubric sensitivity, metric identifiability, and aggregation effects.

Task Definition and Dataset Provenance

The JF-ICR benchmark requires classifiers to annotate corporate responses in IR Q&A exchanges along a five-point ordinal scale, ranging from explicit refusals to strongly time-bound commitments. The dataset used is the test split of TheFinAI/JF-ICR (N=253), parsed without error and validated via SHA-256 checksum to ensure artifact provenance.

Class imbalance is pronounced in this dataset, with +1 (weak commitment) accounting for 55% of cases and the rarest class (−2, strong refusal) comprising less than 1% of responses. Such imbalance accentuates the importance of metric selection, as it directly impacts both discriminative power and robustness of evaluation results.

Measurement Risk: Rubric and Metric Sensitivity

Rubric-Induced Label Sensitivity

Empirical analysis demonstrates substantial label movement attributable to rubric variation, particularly between literal and pragmatic interpretations. Agreement rates between the literal and pragmatic rubric variants averaged 70–83% across models, with dominant disagreement focused on the +1/0 boundary—precisely where pragmatic and literal readings diverge in Japanese business language. A striking finding is that approximately 27.5% of items labeled as +1 (weak commitment) under the literal rubric are relabeled as 0 (neutral/hedged) when pragmatic intent is prioritized. This magnitude of shift underscores that rubric is not merely a superficial knob, but a meaningful determinant of evaluation outcomes.

Metric Identifiability Under Class Imbalance

The metric audit reveals that only exact accuracy, macro-F1, and weighted κ are reliably identified metrics for model ranking within JF-ICR. Within-one accuracy is dominated by the majority class, yielding minimal headroom for meaningful discrimination. Worst-class accuracy is rendered unstable due to the rarity of strong refusals, resulting in excessive noise relative to signal. The authors propose a deterministic metric identifiability checklist, prespecifying thresholds on baseline headroom, class support, and signal-to-noise ratio; metrics failing any threshold are relegated to diagnostic status.

The procedural audit eliminates post-hoc metric exclusion and ensures that the ranking evidence is solely based on metrics that exhibit robust discriminative power given the observed data distribution.

Aggregation and Ranking Stability

Three standard rank-aggregation methods—Bradley-Terry, Borda count, and Ranked Pairs—are applied to both the clean identifiable-metric subset and the full metric sweep. On the clean subset, all three aggregators produce identical model rankings, with claude-sonnet-4-6 as a Condorcet winner and clear tiering among the remaining models. The full five-metric sweep introduces disagreement among aggregators, but leave-one-metric-out analysis demonstrates that either within-one accuracy or worst-class accuracy is sufficient to break this consensus. The audit thus isolates metric degeneracy, not social-choice pathology, as the principal driver of ranking instability within this setting.

CI-significant policy-induced inversions are rare, and only pairs involving the weakest model (Qwen3-235b) yield Holm-surviving differences at the policy level. Dimension sensitivity analysis using Kendall τ-distance confirms that rubric variation contributes the largest share of ranking movement, followed by aggregation metric, and temperature.

Implications for Financial NLP Deployment

The findings have substantial implications for the governance of financial NLP system evaluation and downstream procurement or compliance workflows. The instability introduced by rubric and metric choice mandates operational reporting discipline: every leaderboard should specify rubric text, class distribution, metric diagnostics, uncertainty intervals, and artifact provenance.

The audit procedure is most transferable to settings where label scales are ordinal or asymmetric, class imbalance is prominent, rubric encoding is judgmental, and benchmark outcomes have operational impact. Examples include earnings-call stance analysis, disclosure surveillance, ESG tagging, and covenant monitoring.

Theoretical Contributions and Limitations

The paper advances the formalization of measurement risk in AI evaluation, aligned with representational measurement theory. It operationalizes metric identifiability audits to safeguard evidence, decouples rubric framing from linguistic validation, and mechanistically verifies aggregation stability. However, the experiment’s scope is bounded: generalization claims regarding universal LLM ranking behavior or downstream decision improvements are unsupported. Significant class imbalance and modest sample size (N=253) further constrain the resolution and repeatability of fine-grained ranking claims.

Rubric and framing effects are confounded by the current design, precluding causal isolation of Japanese pragmatic effects from surface-level semantic and verbosity variation. Finally, dataset provenance issues highlight the necessity of rigorous artifact governance to avoid measurement drift.

Speculation on Future Developments

Future work should pursue generalization to culturally heterogeneous benchmarks with larger and more balanced class distributions, integrate robust uncertainty quantification across rubric-authoring variance and model stochasticity, and execute confirmatory replication in settings where Condorcet cycles have been observed.

Further, development of factorial rubric designs and bilingual adjudication can advance causal linguistic validation, improving the interpretability of pragmatic boundaries in financial discourse. Integration with downstream workflows—e.g., automated compliance, risk alerting, or investor-sentiment dashboards—will necessitate increased transparency and rigorous measurement audits prior to operational adoption.

Conclusion

The paper establishes that supervised financial NLP benchmarks entail significant measurement risk due to rubric, metric, and aggregation sensitivity. Gold labels alone do not guarantee objectivity unless the full evaluation pipeline is auditable and robustly governed. For decision evidence in deployment or procurement, reporting discipline must encompass rubric text, metric diagnostics, artifact provenance, and claim scope. The procedural audit presented is the principal contribution, delineating the boundary where evaluation evidence becomes measurement-supported in financial AI. Such discipline is necessary for benchmarks poised to affect high-stakes operational decisions.