Determine which span metric better reflects clinical correctness

Determine whether token F1 or RadGraph F1 more accurately reflects clinical correctness for short extracted spans in radiology question answering, using radiologist adjudication, and consequently determine whether answerable demonstrations primarily improve extraction or answerability.

Background

The paper compares token-level F1 with RadGraph F1, an entity- and relation-oriented metric intended to be less sensitive to span boundaries. The two metrics lead to different conclusions about the effect of answerable demonstrations: token F1 attributes most of the gain to answerability, whereas RadGraph F1 attributes most of it to extraction.

Because no radiologist adjudicated the short answer spans, the paper does not resolve which metric better corresponds to clinical correctness. Resolving this issue would clarify how improvements from prompting should be interpreted in radiology question answering.

References

Without radiologist adjudication we cannot say which better reflects clinical correctness, so we do not claim that answerable demonstrations act mainly through answerability.

— Dissociating Answerability from Extraction in LLM Question Answering over Radiology Reports  (2610.17576 - Sinha et al., 13 Oct 2026) in Section 4.2, paragraph “For answerable demonstrations, the split depends on the metric.”

With about four answerable questions per report, this measure is noisy, which biases the association toward null, so we cannot exclude a modest effect.

— Dissociating Answerability from Extraction in LLM Question Answering over Radiology Reports  (2610.17576 - Sinha et al., 13 Oct 2026) in Section 4.4, paragraph “Within models.”

Here Ext denotes the extractive diagnostic arm; its precise span-selection rule is not established by the available report and is not treated as an oracle.

— What Coding Agents Leave Unsaid: Measuring Task-State Gaps Between Execution and Reports  (2610.20003 - Xu et al., 14 Oct 2026) in Appendix, Section Reporting Interventions, subsection “Report generation and budget”