sbv IMPROVER diagnostic signature challenge

Scoring strategies

Raquel Norel, Erhan Bilal, Nathalie Conrad-Chemineau, Richard Bonneau, Alberto de la Fuente, Igor Jurisica, Daniel Marbach, Pablo Meyer, J. Jeremy Rice, Tamir Tuller, Gustavo Stolovitzky

Research output: Contribution to journalArticle

Abstract

Evaluating the performance of computational methods to analyze high throughput data are an integral component of model development and critical to progress in computational biology. In collaborative-competitions, model performance evaluation is crucial to determine the best performing submission. Here we present the scoring methodology used to assess 54 submissions to the IMPROVER Diagnostic Signature Challenge. Participants were tasked with classifying patients' disease phenotype based on gene expression data in four disease areas: Psoriasis, Chronic Obstructive Pulmonary Disease, Lung Cancer, and Multiple Sclerosis. We discuss the criteria underlying the choice of the three scoring metrics we chose to assess the performance of the submitted models. The statistical significance of the difference in performance between individual submissions and classification tasks varied according to these different metrics. Accordingly, we consider an aggregation of these three assessment methods and present the approaches considered for aggregating the ranking and ultimately determining the final overall best performer.

Original languageEnglish (US)
Pages (from-to)208-216
Number of pages9
JournalSystems Biomedicine
Volume1
Issue number4
DOIs
StatePublished - Jan 1 2014

Fingerprint

Computational Biology
Psoriasis
Chronic Obstructive Pulmonary Disease
Multiple Sclerosis
Lung Neoplasms
Pulmonary diseases
Computational methods
Phenotype
Gene Expression
Gene expression
Agglomeration
Throughput

Keywords

  • Crowdsourcing
  • Diagnostic signature
  • Gene expression
  • Improver
  • Molecular classification
  • Peer-review

ASJC Scopus subject areas

  • Biochemistry
  • Biotechnology
  • Genetics
  • Genetics(clinical)
  • Cell Biology
  • Medicine (miscellaneous)

Cite this

Norel, R., Bilal, E., Conrad-Chemineau, N., Bonneau, R., de la Fuente, A., Jurisica, I., ... Stolovitzky, G. (2014). sbv IMPROVER diagnostic signature challenge: Scoring strategies. Systems Biomedicine, 1(4), 208-216. https://doi.org/10.4161/sysb.26326

sbv IMPROVER diagnostic signature challenge : Scoring strategies. / Norel, Raquel; Bilal, Erhan; Conrad-Chemineau, Nathalie; Bonneau, Richard; de la Fuente, Alberto; Jurisica, Igor; Marbach, Daniel; Meyer, Pablo; Rice, J. Jeremy; Tuller, Tamir; Stolovitzky, Gustavo.

In: Systems Biomedicine, Vol. 1, No. 4, 01.01.2014, p. 208-216.

Research output: Contribution to journalArticle

Norel, R, Bilal, E, Conrad-Chemineau, N, Bonneau, R, de la Fuente, A, Jurisica, I, Marbach, D, Meyer, P, Rice, JJ, Tuller, T & Stolovitzky, G 2014, 'sbv IMPROVER diagnostic signature challenge: Scoring strategies', Systems Biomedicine, vol. 1, no. 4, pp. 208-216. https://doi.org/10.4161/sysb.26326
Norel R, Bilal E, Conrad-Chemineau N, Bonneau R, de la Fuente A, Jurisica I et al. sbv IMPROVER diagnostic signature challenge: Scoring strategies. Systems Biomedicine. 2014 Jan 1;1(4):208-216. https://doi.org/10.4161/sysb.26326
Norel, Raquel ; Bilal, Erhan ; Conrad-Chemineau, Nathalie ; Bonneau, Richard ; de la Fuente, Alberto ; Jurisica, Igor ; Marbach, Daniel ; Meyer, Pablo ; Rice, J. Jeremy ; Tuller, Tamir ; Stolovitzky, Gustavo. / sbv IMPROVER diagnostic signature challenge : Scoring strategies. In: Systems Biomedicine. 2014 ; Vol. 1, No. 4. pp. 208-216.
@article{d1697c5bc3954f51a2598c079687a07b,
title = "sbv IMPROVER diagnostic signature challenge: Scoring strategies",
abstract = "Evaluating the performance of computational methods to analyze high throughput data are an integral component of model development and critical to progress in computational biology. In collaborative-competitions, model performance evaluation is crucial to determine the best performing submission. Here we present the scoring methodology used to assess 54 submissions to the IMPROVER Diagnostic Signature Challenge. Participants were tasked with classifying patients' disease phenotype based on gene expression data in four disease areas: Psoriasis, Chronic Obstructive Pulmonary Disease, Lung Cancer, and Multiple Sclerosis. We discuss the criteria underlying the choice of the three scoring metrics we chose to assess the performance of the submitted models. The statistical significance of the difference in performance between individual submissions and classification tasks varied according to these different metrics. Accordingly, we consider an aggregation of these three assessment methods and present the approaches considered for aggregating the ranking and ultimately determining the final overall best performer.",
keywords = "Crowdsourcing, Diagnostic signature, Gene expression, Improver, Molecular classification, Peer-review",
author = "Raquel Norel and Erhan Bilal and Nathalie Conrad-Chemineau and Richard Bonneau and {de la Fuente}, Alberto and Igor Jurisica and Daniel Marbach and Pablo Meyer and Rice, {J. Jeremy} and Tamir Tuller and Gustavo Stolovitzky",
year = "2014",
month = "1",
day = "1",
doi = "10.4161/sysb.26326",
language = "English (US)",
volume = "1",
pages = "208--216",
journal = "Systems Biomedicine",
issn = "2162-8130",
publisher = "Taylor and Francis Ltd.",
number = "4",

}

TY - JOUR

T1 - sbv IMPROVER diagnostic signature challenge

T2 - Scoring strategies

AU - Norel, Raquel

AU - Bilal, Erhan

AU - Conrad-Chemineau, Nathalie

AU - Bonneau, Richard

AU - de la Fuente, Alberto

AU - Jurisica, Igor

AU - Marbach, Daniel

AU - Meyer, Pablo

AU - Rice, J. Jeremy

AU - Tuller, Tamir

AU - Stolovitzky, Gustavo

PY - 2014/1/1

Y1 - 2014/1/1

N2 - Evaluating the performance of computational methods to analyze high throughput data are an integral component of model development and critical to progress in computational biology. In collaborative-competitions, model performance evaluation is crucial to determine the best performing submission. Here we present the scoring methodology used to assess 54 submissions to the IMPROVER Diagnostic Signature Challenge. Participants were tasked with classifying patients' disease phenotype based on gene expression data in four disease areas: Psoriasis, Chronic Obstructive Pulmonary Disease, Lung Cancer, and Multiple Sclerosis. We discuss the criteria underlying the choice of the three scoring metrics we chose to assess the performance of the submitted models. The statistical significance of the difference in performance between individual submissions and classification tasks varied according to these different metrics. Accordingly, we consider an aggregation of these three assessment methods and present the approaches considered for aggregating the ranking and ultimately determining the final overall best performer.

AB - Evaluating the performance of computational methods to analyze high throughput data are an integral component of model development and critical to progress in computational biology. In collaborative-competitions, model performance evaluation is crucial to determine the best performing submission. Here we present the scoring methodology used to assess 54 submissions to the IMPROVER Diagnostic Signature Challenge. Participants were tasked with classifying patients' disease phenotype based on gene expression data in four disease areas: Psoriasis, Chronic Obstructive Pulmonary Disease, Lung Cancer, and Multiple Sclerosis. We discuss the criteria underlying the choice of the three scoring metrics we chose to assess the performance of the submitted models. The statistical significance of the difference in performance between individual submissions and classification tasks varied according to these different metrics. Accordingly, we consider an aggregation of these three assessment methods and present the approaches considered for aggregating the ranking and ultimately determining the final overall best performer.

KW - Crowdsourcing

KW - Diagnostic signature

KW - Gene expression

KW - Improver

KW - Molecular classification

KW - Peer-review

UR - http://www.scopus.com/inward/record.url?scp=84984671808&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84984671808&partnerID=8YFLogxK

U2 - 10.4161/sysb.26326

DO - 10.4161/sysb.26326

M3 - Article

VL - 1

SP - 208

EP - 216

JO - Systems Biomedicine

JF - Systems Biomedicine

SN - 2162-8130

IS - 4

ER -