- The paper shows that financial NER accuracy and confidence reliability degrade sharply under domain shift, with encoder F1 falling from 70.3% in-domain to 24.1% on tweets.
- The study finds that sequence probability performs best in-domain, while span probability and self-consistency rank errors more reliably under shift, although self-consistency can remain confidently wrong.
- Selective prediction greatly reduces errors in supported domains, but fails under extreme shift, supporting a deployment strategy that combines shift detection, confidence gating, and human review.
Motivation and problem setting
Financial named entity recognition (NER) is typically evaluated with F1 on a single in-domain split, but deployed extractors must process filings, news, and social media. The paper asks a deployment-oriented question that leaderboard F1 cannot answer: when the input distribution drifts from the training data, can the model identify which extractions are trustworthy? This matters because neural NER models lean heavily on memorized entity surface forms and become overconfident precisely under shift (2608.19558). In financial pipelines, a hallucinated organization or mistyped person entity propagates into compliance, monitoring, and analytical workflows.
The study frames the task as calibrated selective prediction: models abstain on low-confidence predictions for routing to human review, trading coverage for reduced risk. Models — bert-base-cased taggers and LoRA-tuned Qwen2.5-0.5B/1.5B-Instruct generators trained on the FIN corpus of SEC filings (~1k sentences) — face a three-tier stress test: in-domain FIN test, near-shift FiNER-ORD (financial news), and far-shift TweetNER7 (general-topic tweets) as an extreme out-of-domain boundary condition. All thresholds and calibration parameters were fit on the FIN validation split and frozen before any test evaluation, with three training seeds and 1,000-resample bootstrap intervals.
A notable methodological choice: TweetNER7 is not a financial corpus; it is used deliberately as an extreme shift probe. The main protocol drops tweets containing out-of-schema entities (2,179 of 2,807 in the 2021 split), leaving 628 eligible sentences capped at 300; a sensitivity analysis confirms headline conclusions hold under the relaxed ignore-entity protocol.
The encoder tagger achieves 70.3 micro-F1 in-domain but falls to 38.3 on news and 24.1 on tweets — a two-thirds relative loss. The 0.5B generative model starts far weaker in-domain (37.4±1.3 F1; encoder advantage +33.6 points, bootstrap CI [22.7, 44.5]), but degrades more gently: on the far tier its advantage vanishes entirely (24.1±2.5 vs. 26.7±2.8), consistent with prior reports that instruction-tuned generative models generalize more gracefully across domains. Generation-specific failures remain bounded: 1.4–5.4% invalid JSON and 3.6–8.2% hallucinated entities per tier. Because encoders and generators are scored under different matching procedures (exact BIO spans vs. position-less multiset matching), direct cross-family F1 comparisons are indicative rather than exact — a caveat the authors state explicitly.
A single-seed 1.5B scale check shows substantially better robustness (F1 46.2/52.1/47.7 across tiers; far-tier sentence error rate 86% vs. ~95% for 0.5B), though the authors caution this is qualitative, not a demonstrated replication.
Confidence signal rankings change under shift
Three regularities emerge from comparing five inference-time confidence signals (sequence probability, token probability, span probability, type probability, and self-consistency over K=5 sampled decodes at temperature 0.7):
The best in-domain detector is not the best detector under shift. Sequence probability leads in-domain (AUROC 0.839±0.014) but collapses to 0.661/0.626 on news/tweets, while span probability holds at 0.690/0.721 and self-consistency at 0.671/0.682. On the far tier, sequence- and span-probability bootstrap CIs separate ([0.594, 0.656] vs. [0.695, 0.748]), so the reordering is not sampling noise. This refutes, under shift, prior cautions that simple probability baselines are hard to beat.
Discrimination and calibration dissociate. Probability signals are severely miscalibrated (span-signal ECE 0.36–0.41), whereas self-consistency vote share is well calibrated without any post-hoc fitting (ECE 0.10–0.12). However, reliability diagrams show that on both shifted tiers the highest-confidence bin of self-consistency inverts: the model can be consistently, confidently wrong. Any threshold-based automation policy relying on high-confidence regions must therefore be validated domain by domain.
The encoder's confidence collapses; the generative model's does not. The BERT tagger's MSP is excellent in-domain (AUROC 0.922±0.016) but drops to 0.633/0.615 under shift — mirroring Kamath et al.'s findings for selective QA — while generative span confidence loses far less (0.801→0.690/0.721). Under severe shift, the weaker model provides the more useful uncertainty estimate.
The mechanistic explanation offered for sequence probability's fragility is dilution by output scaffolding: once JSON format is learned, structural tokens receive probability near 1 regardless of input domain, compressing whole-output scores (mean sequence probability exceeds 0.9 on tweets even for wrong predictions). Span-restricted confidence excludes this scaffolding. Consistently, type confidence contributes almost no discriminative value (AUROC 0.49–0.54); for these models, uncertainty resides in where the entity is, not its type.
What abstention buys, and where it stops working
Selective prediction is highly effective within supported domains. Answering only the top-40%-confidence in-domain sentences reduces sentence-level error from 34.3% to below 2% on every seed, and the encoder reaches 98.7% entity precision at 50% coverage (from 66.7% at full coverage). On financial news, abstention remains useful: sentence risk falls from 58.8% toward ~15–25%, and span-ranked selective precision rises +10 points at 60% coverage (30.7→40.8). On the far tier, however, with ~95% of sentences containing at least one error, no signal recovers a usefully large clean subset (AURC 0.92–0.95). Abstention complements, but cannot substitute for, domain-appropriate training data.
Two additional findings carry deployment weight. First, hallucinated entities skew toward low confidence — 67% of hallucinated spans on tweets fall in the bottom span-confidence tertile versus 33% expected uniformly — so a confidence gate preferentially removes fabricated entities, the highest-risk error class in financial extraction. Second, manual inspection attributes the far-tier failure to surface-form bias: the model consistently extracts salient capitalized tokens as ORG/PER across all five samples, illustrating that self-consistency measures stability of a bias rather than correctness.
These results jointly motivate a staged deployment policy: detect severe distribution shift upstream, apply confidence-gated selective prediction only within acceptable domains, and route severe-shift inputs directly to human review. Notably, the paper implements and evaluates none of these stages' first component; distribution-level harmful-covariate-shift tests [ginsberg2023harmful] are identified as the natural candidate, and benchmarking such routing is left as future work.
Limitations and open questions
The authors concede several constraints. Scale is modest by production standards (0.5B/1.5B models, ~1k training sentences), making absolute F1 values lower bounds, and whether shift-degradation patterns attenuate at much larger scale remains open. The far-shift tier conflates register, topic, annotation process, time period, and entity density, so the tiers do not constitute a controlled causal decomposition of domain shift; a matched, human-annotated financial social-media test set would be the definitive fix. ECE estimates on ~300-sentence tiers are unstable and can conceal high-confidence failures, which motivates adaptive or classwise calibration analyses. Verbalized confidence and P(True)-style self-verification are excluded, semantic entropy is only approximated via surface-form vote share, entity-level handoff and learned deferral are unimplemented, and the upstream domain detector the results motivate is not built here.
Conclusion
This work reframes financial NER as a reliability question rather than purely an extraction benchmark. Its central empirical claims — that confidence signal rankings invert under shift, that span probability and self-consistency offer complementary strengths (error ranking vs. calibration), and that selective prediction nearly eliminates risk in-domain but fails outright under extreme shift — are supported across three seeds and qualitatively reproduced at 1.5B scale. The practical takeaway is a staged architecture combining upstream shift detection, within-domain confidence gating, and human review of out-of-distribution inputs; validating and benchmarking the shift-detection stage is the most immediate open question this study leaves unresolved.