Assess inter-rater reliability and elicitation sensitivity

Assess the inter-rater reliability of the structured gain-function elicitation procedure and determine how sensitive the optimised graphical multiple testing procedure is to variability in the elicited stakeholder scores.

Background

The paper proposes eliciting the relative value of incremental label claims from a cross-functional panel of clinical, commercial, and regulatory stakeholders. Individual ratings are aggregated, typically using the median of final-round scores, and the resulting gain function is then used to optimise the graphical multiple testing procedure.

The authors note that the additive gain model simplifies elicitation but may not capture diminishing returns or other dependencies among label claims. They explicitly identify two unresolved methodological issues: the reliability of agreement among raters and the extent to which differences or uncertainty in elicited scores alter the graph selected by the optimisation.