Options
Unsupervised Crowdsourcing with Accuracy and Cost Guarantees
Journal
2022 20th International Symposium on Modeling and Optimization in Mobile, Ad hoc, and Wireless Networks, WiOpt 2022
Date Issued
2022-01-01
Author(s)
Didwania, Yashvardhan
Nair, Jayakrishnan
Hemachandra, N.
Abstract
We consider the problem of cost-optimal utilization of a crowdsourcing platform for binary, unsupervised classification of a collection of items, given a prescribed error threshold. Workers on the crowdsourcing platform are assumed to be divided into multiple classes, based on their skill, experience, and/or past performance. We model each worker class via an unknown confusion matrix, and a (known) price to be paid per label prediction. For this setting, we propose algorithms for acquiring label predictions from workers, and for inferring the true labels of items. We prove that (i) our algorithms satisfy the prescribed error threshold, and (ii) if the number of (unlabeled) items available is large enough, the algorithms incur a cost that is near-optimal. Finally, we validate our algorithms, and some heuristics inspired by them, through an extensive case study.