Crowdsourcing Thousands of Specialized Labels: A Bayesian Active Training Approach

Maximilien Servajean, Alexis Joly, Dennis Shasha, Julien Champ, Esther Pacitti

Research output: Contribution to journalArticle

Abstract

Large-scale annotated corpora have yielded impressive performance improvements in computer vision and multimedia content analysis. However, such datasets depend on an enormous amount of human labeling effort. When the labels correspond to well-known concepts, it is straightforward to train the annotators by giving a few examples with known answers. It is also straightforward to judge the quality of their labels. Neither is true when there are thousands of complex domain-specific labels. Training on all labels is infeasible and the quality of an annotator's judgements may be vastly different for some subsets of labels than for others. This paper proposes a set of data-driven algorithms to 1) train image annotators on how to disambiguate among automatically generated candidate labels, 2) evaluate the quality of annotators' label suggestions, and 3) weigh predictions. The algorithms adapt to the skills of each annotator both in the questions asked and the weights given to their answers. The underlying judgements are Bayesian, based on adaptive priors. We measure the benefits of these algorithms on a live user experiment related to image-based plant identification involving around 1000 people. The proposed methods are shown to enable huge gains in annotation accuracy. A standard user can correctly label around 2% of our data. This goes up to 80% with machine learning assisted training and assignment and up to almost 90% when doing a weighted combination of several annotators' labels.

Original languageEnglish (US)
Article number7819540
Pages (from-to)1376-1391
Number of pages16
JournalIEEE Transactions on Multimedia
Volume19
Issue number6
DOIs
StatePublished - Jun 1 2017

Fingerprint

Labels
Labeling
Computer vision
Learning systems

Keywords

  • Bayes methods
  • Crowdsourcing
  • parameter estimation
  • Taylor series

ASJC Scopus subject areas

  • Signal Processing
  • Media Technology
  • Computer Science Applications
  • Electrical and Electronic Engineering

Cite this

Crowdsourcing Thousands of Specialized Labels : A Bayesian Active Training Approach. / Servajean, Maximilien; Joly, Alexis; Shasha, Dennis; Champ, Julien; Pacitti, Esther.

In: IEEE Transactions on Multimedia, Vol. 19, No. 6, 7819540, 01.06.2017, p. 1376-1391.

Research output: Contribution to journalArticle

Servajean, Maximilien ; Joly, Alexis ; Shasha, Dennis ; Champ, Julien ; Pacitti, Esther. / Crowdsourcing Thousands of Specialized Labels : A Bayesian Active Training Approach. In: IEEE Transactions on Multimedia. 2017 ; Vol. 19, No. 6. pp. 1376-1391.
@article{cc623c54d6ab42588a9038f02f20d713,
title = "Crowdsourcing Thousands of Specialized Labels: A Bayesian Active Training Approach",
abstract = "Large-scale annotated corpora have yielded impressive performance improvements in computer vision and multimedia content analysis. However, such datasets depend on an enormous amount of human labeling effort. When the labels correspond to well-known concepts, it is straightforward to train the annotators by giving a few examples with known answers. It is also straightforward to judge the quality of their labels. Neither is true when there are thousands of complex domain-specific labels. Training on all labels is infeasible and the quality of an annotator's judgements may be vastly different for some subsets of labels than for others. This paper proposes a set of data-driven algorithms to 1) train image annotators on how to disambiguate among automatically generated candidate labels, 2) evaluate the quality of annotators' label suggestions, and 3) weigh predictions. The algorithms adapt to the skills of each annotator both in the questions asked and the weights given to their answers. The underlying judgements are Bayesian, based on adaptive priors. We measure the benefits of these algorithms on a live user experiment related to image-based plant identification involving around 1000 people. The proposed methods are shown to enable huge gains in annotation accuracy. A standard user can correctly label around 2{\%} of our data. This goes up to 80{\%} with machine learning assisted training and assignment and up to almost 90{\%} when doing a weighted combination of several annotators' labels.",
keywords = "Bayes methods, Crowdsourcing, parameter estimation, Taylor series",
author = "Maximilien Servajean and Alexis Joly and Dennis Shasha and Julien Champ and Esther Pacitti",
year = "2017",
month = "6",
day = "1",
doi = "10.1109/TMM.2017.2653763",
language = "English (US)",
volume = "19",
pages = "1376--1391",
journal = "IEEE Transactions on Multimedia",
issn = "1520-9210",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
number = "6",

}

TY - JOUR

T1 - Crowdsourcing Thousands of Specialized Labels

T2 - A Bayesian Active Training Approach

AU - Servajean, Maximilien

AU - Joly, Alexis

AU - Shasha, Dennis

AU - Champ, Julien

AU - Pacitti, Esther

PY - 2017/6/1

Y1 - 2017/6/1

N2 - Large-scale annotated corpora have yielded impressive performance improvements in computer vision and multimedia content analysis. However, such datasets depend on an enormous amount of human labeling effort. When the labels correspond to well-known concepts, it is straightforward to train the annotators by giving a few examples with known answers. It is also straightforward to judge the quality of their labels. Neither is true when there are thousands of complex domain-specific labels. Training on all labels is infeasible and the quality of an annotator's judgements may be vastly different for some subsets of labels than for others. This paper proposes a set of data-driven algorithms to 1) train image annotators on how to disambiguate among automatically generated candidate labels, 2) evaluate the quality of annotators' label suggestions, and 3) weigh predictions. The algorithms adapt to the skills of each annotator both in the questions asked and the weights given to their answers. The underlying judgements are Bayesian, based on adaptive priors. We measure the benefits of these algorithms on a live user experiment related to image-based plant identification involving around 1000 people. The proposed methods are shown to enable huge gains in annotation accuracy. A standard user can correctly label around 2% of our data. This goes up to 80% with machine learning assisted training and assignment and up to almost 90% when doing a weighted combination of several annotators' labels.

AB - Large-scale annotated corpora have yielded impressive performance improvements in computer vision and multimedia content analysis. However, such datasets depend on an enormous amount of human labeling effort. When the labels correspond to well-known concepts, it is straightforward to train the annotators by giving a few examples with known answers. It is also straightforward to judge the quality of their labels. Neither is true when there are thousands of complex domain-specific labels. Training on all labels is infeasible and the quality of an annotator's judgements may be vastly different for some subsets of labels than for others. This paper proposes a set of data-driven algorithms to 1) train image annotators on how to disambiguate among automatically generated candidate labels, 2) evaluate the quality of annotators' label suggestions, and 3) weigh predictions. The algorithms adapt to the skills of each annotator both in the questions asked and the weights given to their answers. The underlying judgements are Bayesian, based on adaptive priors. We measure the benefits of these algorithms on a live user experiment related to image-based plant identification involving around 1000 people. The proposed methods are shown to enable huge gains in annotation accuracy. A standard user can correctly label around 2% of our data. This goes up to 80% with machine learning assisted training and assignment and up to almost 90% when doing a weighted combination of several annotators' labels.

KW - Bayes methods

KW - Crowdsourcing

KW - parameter estimation

KW - Taylor series

UR - http://www.scopus.com/inward/record.url?scp=85028344775&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85028344775&partnerID=8YFLogxK

U2 - 10.1109/TMM.2017.2653763

DO - 10.1109/TMM.2017.2653763

M3 - Article

AN - SCOPUS:85028344775

VL - 19

SP - 1376

EP - 1391

JO - IEEE Transactions on Multimedia

JF - IEEE Transactions on Multimedia

SN - 1520-9210

IS - 6

M1 - 7819540

ER -