Quality Enhancement by Weighted Rank Aggregation of Crowd Opinion

Abstract: Expertise of annotators has a major role in crowdsourcing based opinion aggregation models. In such frameworks, accuracy and biasness of annotators are occasionally taken as important features and based on them priority of the annotators are assigned. But instead of relying on a single feature, multiple features can be considered and separate rankings can be produced to judge the annotators properly. Finally, the aggregation of those rankings with perfect weightage can be done with an aim to produce better ground truth prediction. Here, we propose a novel weighted rank aggregation method and its efficacy with respect to other existing approaches is shown on artificial dataset. The effectiveness of weighted rank aggregation to enhance quality prediction is also shown by applying it on an Amazon Mechanical Turk (AMT) dataset.