Subcomponent-level entailment scoring for overinclusive and partial answers

Develop a subcomponent-level entailment-scoring method that resolves the bidirectional-NLI artifact causing confusion between overinclusive-valid and partial answers in the CAP framework for open-ended question answering.

Background

The CAP metric evaluates question-conditioned declarative statements using bidirectional natural-language inference. The paper’s error analysis shows that this design can assign excessively high scores to partial answers because the gold statement entails the partial statement, while assigning comparatively low scores to overinclusive-valid answers whose additional valid information is not entailed by the gold statement. The authors identify scoring entailment over smaller answer subcomponents as an unresolved direction that could expose and correct this asymmetry.

References

Several directions remain open for future work. Splitting answers into smaller subcomponents and scoring entailment over each could resolve the bidirectional-NLI artefact on the overinclusive-valid / partial axis; stronger NLI backbones with broader world knowledge could close the alternative-correct ceiling; and multilingual NLI checkpoints could port the framework off English.

— How Correct Is Your Answer? A Semantic Correctness Framework for Open QA Evaluation  (2609.01369 - Yotkova et al., 1 Sep 2026) in Section 7, Conclusion and Future Work

Several directions remain open for future work. Splitting answers into smaller subcomponents and scoring entailment over each could resolve the bidirectional-NLI artefact on the overinclusive-valid / partial axis; stronger NLI backbones with broader world knowledge could close the alternative-correct ceiling; and multilingual NLI checkpoints could port the framework off English.

— How Correct Is Your Answer? A Semantic Correctness Framework for Open QA Evaluation  (2609.01369 - Yotkova et al., 1 Sep 2026) in Section 7, Conclusion and Future Work