Determine how active-learning and anytime-valid inference methods apply to complex clinical abstraction

Determine how active-learning and safe anytime-valid inference techniques can be applied to highly complex patient notes and clinical data abstraction tasks to structure adaptive, user-guided validation workflows.

Background

Libretto users preferred adaptive, ad-hoc evaluation in which they inspected small groups of patients, searched for likely failure modes, and refined specifications iteratively. Although this approach was practical, it made rigorous accuracy estimation difficult. The paper identifies active learning and safe anytime-valid inference as possible frameworks for structuring such workflows, while explicitly leaving their application to complex clinical notes and abstraction tasks unresolved.

References

Some recent work has explored how frameworks such as active learning and safe anytime-valid inference can be used to structure these adaptive workflows; however, future work is needed to understand how these techniques can be applied to highly complex patient notes and abstraction tasks.

— "I Know Where to Look," But Does the LLM? Charting the Gaps Between Clinical Expert Needs and Unstructured Data Abstraction Tools  (2609.19318 - Sivaraman et al., 16 Sep 2026) in Section 6.2, “Future Design Directions,” subsection “Evaluating Results,” item D7 (“Active, user-guided validation”)