Supervision under richer reliability parameterisations

Determine whether supervised reliability estimation outperforms label-free estimation when expert reliabilities are parameterised more richly, such as per subject or conditional on the query.

Background

The paper estimates one reliability per expert and compares the resulting label-free EM-based jury with a supervised linear fusion whose reliabilities are derived from gold-label accuracy. It then relaxes the one-reliability-per-expert assumption by fitting separate reliabilities for each task. Both supervised and label-free approaches improve under this richer parameterisation, but the label-free jury remains ahead in the reported benchmark.

The unresolved issue is whether this observed advantage persists—or reverses—in a broader evaluation with richer reliability models, including reliabilities defined per subject or conditioned on the query. The paper explicitly leaves the relative benefit of supervision under such parameterisations unresolved.