Machine-crowd-expert model for increasing user engagement and annotation quality

Ana Elisa Méndez Méndez, Mark Cartwright, Juan Bello

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Crowdsourcing and active learning have been combined in the past with the goal of reducing annotation costs. However, two issues arise with using AL and crowdsourcing: quality of the labels and user engagement. In this work, we propose a novel machine ⇔ crowd ⇔ expert loop model where the forward connections of the loop aim to improve the quality of the labels and the backward connections aim to increase user engagement. In addition, we propose a research agenda for evaluating the model.

Original languageEnglish (US)
Title of host publicationCHI EA 2019 - Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems
PublisherAssociation for Computing Machinery
ISBN (Electronic)9781450359719
DOIs
StatePublished - May 2 2019
Event2019 CHI Conference on Human Factors in Computing Systems, CHI EA 2019 - Glasgow, United Kingdom
Duration: May 4 2019May 9 2019

Publication series

NameConference on Human Factors in Computing Systems - Proceedings

Conference

Conference2019 CHI Conference on Human Factors in Computing Systems, CHI EA 2019
CountryUnited Kingdom
CityGlasgow
Period5/4/195/9/19

Fingerprint

Labels
Costs
Problem-Based Learning

Keywords

  • Active learning
  • Crowdsourcing
  • Sound classification

ASJC Scopus subject areas

  • Human-Computer Interaction
  • Computer Graphics and Computer-Aided Design
  • Software

Cite this

Méndez, A. E. M., Cartwright, M., & Bello, J. (2019). Machine-crowd-expert model for increasing user engagement and annotation quality. In CHI EA 2019 - Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems [3313054] (Conference on Human Factors in Computing Systems - Proceedings). Association for Computing Machinery. https://doi.org/10.1145/3290607.3313054

Machine-crowd-expert model for increasing user engagement and annotation quality. / Méndez, Ana Elisa Méndez; Cartwright, Mark; Bello, Juan.

CHI EA 2019 - Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, 2019. 3313054 (Conference on Human Factors in Computing Systems - Proceedings).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Méndez, AEM, Cartwright, M & Bello, J 2019, Machine-crowd-expert model for increasing user engagement and annotation quality. in CHI EA 2019 - Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems., 3313054, Conference on Human Factors in Computing Systems - Proceedings, Association for Computing Machinery, 2019 CHI Conference on Human Factors in Computing Systems, CHI EA 2019, Glasgow, United Kingdom, 5/4/19. https://doi.org/10.1145/3290607.3313054
Méndez AEM, Cartwright M, Bello J. Machine-crowd-expert model for increasing user engagement and annotation quality. In CHI EA 2019 - Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery. 2019. 3313054. (Conference on Human Factors in Computing Systems - Proceedings). https://doi.org/10.1145/3290607.3313054
Méndez, Ana Elisa Méndez ; Cartwright, Mark ; Bello, Juan. / Machine-crowd-expert model for increasing user engagement and annotation quality. CHI EA 2019 - Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, 2019. (Conference on Human Factors in Computing Systems - Proceedings).
@inproceedings{3386b6606855484e9fce08b14f93696d,
title = "Machine-crowd-expert model for increasing user engagement and annotation quality",
abstract = "Crowdsourcing and active learning have been combined in the past with the goal of reducing annotation costs. However, two issues arise with using AL and crowdsourcing: quality of the labels and user engagement. In this work, we propose a novel machine ⇔ crowd ⇔ expert loop model where the forward connections of the loop aim to improve the quality of the labels and the backward connections aim to increase user engagement. In addition, we propose a research agenda for evaluating the model.",
keywords = "Active learning, Crowdsourcing, Sound classification",
author = "M{\'e}ndez, {Ana Elisa M{\'e}ndez} and Mark Cartwright and Juan Bello",
year = "2019",
month = "5",
day = "2",
doi = "10.1145/3290607.3313054",
language = "English (US)",
series = "Conference on Human Factors in Computing Systems - Proceedings",
publisher = "Association for Computing Machinery",
booktitle = "CHI EA 2019 - Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems",

}

TY - GEN

T1 - Machine-crowd-expert model for increasing user engagement and annotation quality

AU - Méndez, Ana Elisa Méndez

AU - Cartwright, Mark

AU - Bello, Juan

PY - 2019/5/2

Y1 - 2019/5/2

N2 - Crowdsourcing and active learning have been combined in the past with the goal of reducing annotation costs. However, two issues arise with using AL and crowdsourcing: quality of the labels and user engagement. In this work, we propose a novel machine ⇔ crowd ⇔ expert loop model where the forward connections of the loop aim to improve the quality of the labels and the backward connections aim to increase user engagement. In addition, we propose a research agenda for evaluating the model.

AB - Crowdsourcing and active learning have been combined in the past with the goal of reducing annotation costs. However, two issues arise with using AL and crowdsourcing: quality of the labels and user engagement. In this work, we propose a novel machine ⇔ crowd ⇔ expert loop model where the forward connections of the loop aim to improve the quality of the labels and the backward connections aim to increase user engagement. In addition, we propose a research agenda for evaluating the model.

KW - Active learning

KW - Crowdsourcing

KW - Sound classification

UR - http://www.scopus.com/inward/record.url?scp=85067307264&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85067307264&partnerID=8YFLogxK

U2 - 10.1145/3290607.3313054

DO - 10.1145/3290607.3313054

M3 - Conference contribution

AN - SCOPUS:85067307264

T3 - Conference on Human Factors in Computing Systems - Proceedings

BT - CHI EA 2019 - Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems

PB - Association for Computing Machinery

ER -