A Machine Learning-Based Approach to Enhancing Label Aggregation in Crowdsourcing

Abstract

Crowdsourcing is the most effective means of obtaining labelled data for supervised machine learning. However, the varying expertise of crowd workers often results in noisy annotations. While traditional label aggregation methods attempt to handle label noise, they typically overlook the relationships between different data instances. Moreover, crowdsourced datasets often experience class imbalance, wherein predominant classes eclipse minority classes, hence exacerbating label accuracy issues. This paper proposed a Reliability-Weighted Bayesian Label Aggregation (RWBLA) to overcome the above challenges. First, the K-nearest neighbours (KNN) method is used to improve the label set for each instance by augmenting labels from its closest neighbours, resulting in multiple noisy label sets. Next, it improves the aggregation by assigning weights to the neighbouring labels based on worker reliability, adaptive distance, and label similarity for each instance. In addition, worker reliability is determined dynamically based on the neighbourhood information, and to handle the imbalance issue, label similarity is modified. In the end, a weighted Bayesian inference method is used to infer the correct label for each instance. The performance of the proposed approach is evaluated on 20 synthetics and three real-world crowdsourcing datasets. It shows that RWBLA consistently surpasses eight baseline label aggregations, improving aggregation accuracy by 3 % to 13 %. Moreover, in analyses utilizing imbalanced real-world crowdsourcing data, RWBLA surpassed state-of-art aggregation algorithms by 2 % to 6 %, underscoring its efficacy in situations with minor class imbalances.