Reputation-based worker filtering in crowdsourcing

Srikanth Jagabathula, Lakshminarayanan Subramanian, Ashwin Venkataraman

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

In this paper, we study the problem of aggregating noisy labels from crowd workers to infer the underlying true labels of binary tasks. Unlike most prior work which has examined this problem under the random worker paradigm, we consider a much broader class of adversarial workers with no specific assumptions on their labeling strategy. Our key contribution is the design of a computationally efficient reputation algorithm to identify and filter out these adversarial workers in crowd-sourcing systems. Our algorithm uses the concept of optimal semi-matchings in conjunction with worker penalties based on label disagreements, to assign a reputation score for every worker. We provide strong theoretical guarantees for deterministic adversarial strategies as well as the extreme case of sophisticated adversaries where we analyze the worst-case behavior of our algorithm. Finally, we show that our reputation algorithm can significantly improve the accuracy of existing label aggregation algorithms in real-world crowdsourcing datasets.

Original languageEnglish (US)
Title of host publicationAdvances in Neural Information Processing Systems
PublisherNeural information processing systems foundation
Pages2492-2500
Number of pages9
Volume3
EditionJanuary
StatePublished - 2014
Event28th Annual Conference on Neural Information Processing Systems 2014, NIPS 2014 - Montreal, Canada
Duration: Dec 8 2014Dec 13 2014

Other

Other28th Annual Conference on Neural Information Processing Systems 2014, NIPS 2014
CountryCanada
CityMontreal
Period12/8/1412/13/14

Fingerprint

Labels
Labeling
Agglomeration

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Information Systems
  • Signal Processing

Cite this

Jagabathula, S., Subramanian, L., & Venkataraman, A. (2014). Reputation-based worker filtering in crowdsourcing. In Advances in Neural Information Processing Systems (January ed., Vol. 3, pp. 2492-2500). Neural information processing systems foundation.

Reputation-based worker filtering in crowdsourcing. / Jagabathula, Srikanth; Subramanian, Lakshminarayanan; Venkataraman, Ashwin.

Advances in Neural Information Processing Systems. Vol. 3 January. ed. Neural information processing systems foundation, 2014. p. 2492-2500.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Jagabathula, S, Subramanian, L & Venkataraman, A 2014, Reputation-based worker filtering in crowdsourcing. in Advances in Neural Information Processing Systems. January edn, vol. 3, Neural information processing systems foundation, pp. 2492-2500, 28th Annual Conference on Neural Information Processing Systems 2014, NIPS 2014, Montreal, Canada, 12/8/14.
Jagabathula S, Subramanian L, Venkataraman A. Reputation-based worker filtering in crowdsourcing. In Advances in Neural Information Processing Systems. January ed. Vol. 3. Neural information processing systems foundation. 2014. p. 2492-2500
Jagabathula, Srikanth ; Subramanian, Lakshminarayanan ; Venkataraman, Ashwin. / Reputation-based worker filtering in crowdsourcing. Advances in Neural Information Processing Systems. Vol. 3 January. ed. Neural information processing systems foundation, 2014. pp. 2492-2500
@inproceedings{58f81ae45931426b960e91581a4b591c,
title = "Reputation-based worker filtering in crowdsourcing",
abstract = "In this paper, we study the problem of aggregating noisy labels from crowd workers to infer the underlying true labels of binary tasks. Unlike most prior work which has examined this problem under the random worker paradigm, we consider a much broader class of adversarial workers with no specific assumptions on their labeling strategy. Our key contribution is the design of a computationally efficient reputation algorithm to identify and filter out these adversarial workers in crowd-sourcing systems. Our algorithm uses the concept of optimal semi-matchings in conjunction with worker penalties based on label disagreements, to assign a reputation score for every worker. We provide strong theoretical guarantees for deterministic adversarial strategies as well as the extreme case of sophisticated adversaries where we analyze the worst-case behavior of our algorithm. Finally, we show that our reputation algorithm can significantly improve the accuracy of existing label aggregation algorithms in real-world crowdsourcing datasets.",
author = "Srikanth Jagabathula and Lakshminarayanan Subramanian and Ashwin Venkataraman",
year = "2014",
language = "English (US)",
volume = "3",
pages = "2492--2500",
booktitle = "Advances in Neural Information Processing Systems",
publisher = "Neural information processing systems foundation",
edition = "January",

}

TY - GEN

T1 - Reputation-based worker filtering in crowdsourcing

AU - Jagabathula, Srikanth

AU - Subramanian, Lakshminarayanan

AU - Venkataraman, Ashwin

PY - 2014

Y1 - 2014

N2 - In this paper, we study the problem of aggregating noisy labels from crowd workers to infer the underlying true labels of binary tasks. Unlike most prior work which has examined this problem under the random worker paradigm, we consider a much broader class of adversarial workers with no specific assumptions on their labeling strategy. Our key contribution is the design of a computationally efficient reputation algorithm to identify and filter out these adversarial workers in crowd-sourcing systems. Our algorithm uses the concept of optimal semi-matchings in conjunction with worker penalties based on label disagreements, to assign a reputation score for every worker. We provide strong theoretical guarantees for deterministic adversarial strategies as well as the extreme case of sophisticated adversaries where we analyze the worst-case behavior of our algorithm. Finally, we show that our reputation algorithm can significantly improve the accuracy of existing label aggregation algorithms in real-world crowdsourcing datasets.

AB - In this paper, we study the problem of aggregating noisy labels from crowd workers to infer the underlying true labels of binary tasks. Unlike most prior work which has examined this problem under the random worker paradigm, we consider a much broader class of adversarial workers with no specific assumptions on their labeling strategy. Our key contribution is the design of a computationally efficient reputation algorithm to identify and filter out these adversarial workers in crowd-sourcing systems. Our algorithm uses the concept of optimal semi-matchings in conjunction with worker penalties based on label disagreements, to assign a reputation score for every worker. We provide strong theoretical guarantees for deterministic adversarial strategies as well as the extreme case of sophisticated adversaries where we analyze the worst-case behavior of our algorithm. Finally, we show that our reputation algorithm can significantly improve the accuracy of existing label aggregation algorithms in real-world crowdsourcing datasets.

UR - http://www.scopus.com/inward/record.url?scp=84937907918&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84937907918&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:84937907918

VL - 3

SP - 2492

EP - 2500

BT - Advances in Neural Information Processing Systems

PB - Neural information processing systems foundation

ER -