Data-driven rank aggregation with application to grand challenges

James Fishbaugh, Marcel Prastawa, Bo Wang, Patrick Reynolds, Stephen Aylward, Guido Gerig

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Abstract

    The increased number of challenges for comparative evaluation of biomedical image analysis procedures clearly reflects a need for unbiased assessment of the state-of-the-art methodological advances. Moreover, the ultimate translation of novel image analysis procedures to the clinic requires rigorous validation and evaluation of alternative schemes, a task that is best outsourced to the international research community. We commonly see an increase of the number of metrics to be used in parallel, reflecting alternative ways to measure similarity. Since different measures come with different scales and distributions, these are often normalized or converted into an individual rank ordering, leaving the problem of combining the set of multiple rankings into a final score. Proposed solutions are averaging or accumulation of rankings, raising the question if different metrics are to be treated the same or if all metrics would be needed to assess closeness to truth. We address this issue with a data-driven method for automatic estimation of weights for a set of metrics based on unsupervised rank aggregation. Our method requires no normalization procedures and makes no assumptions about metric distributions. We explore the sensitivity of metrics to small changes in input data with an iterative perturbation scheme, to prioritize the contribution of the most robust metrics in the overall ranking. We show on real anatomical data that our weighting scheme can dramatically change the ranking.

    Original languageEnglish (US)
    Title of host publicationMedical Image Computing and Computer Assisted Intervention − MICCAI 2017 - 20th International Conference, Proceedings
    PublisherSpringer Verlag
    Pages754-762
    Number of pages9
    Volume10434 LNCS
    ISBN (Print)9783319661841
    DOIs
    StatePublished - 2017
    Event20th International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2017 - Quebec City, Canada
    Duration: Sep 11 2017Sep 13 2017

    Publication series

    NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
    Volume10434 LNCS
    ISSN (Print)0302-9743
    ISSN (Electronic)1611-3349

    Other

    Other20th International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2017
    CountryCanada
    CityQuebec City
    Period9/11/179/13/17

    Fingerprint

    Rank Aggregation
    Data-driven
    Image analysis
    Agglomeration
    Metric
    Ranking
    Image Analysis
    Alternatives
    Evaluation
    Similarity Measure
    Normalization
    Weighting
    Averaging
    Perturbation

    ASJC Scopus subject areas

    • Theoretical Computer Science
    • Computer Science(all)

    Cite this

    Fishbaugh, J., Prastawa, M., Wang, B., Reynolds, P., Aylward, S., & Gerig, G. (2017). Data-driven rank aggregation with application to grand challenges. In Medical Image Computing and Computer Assisted Intervention − MICCAI 2017 - 20th International Conference, Proceedings (Vol. 10434 LNCS, pp. 754-762). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 10434 LNCS). Springer Verlag. https://doi.org/10.1007/978-3-319-66185-8_85

    Data-driven rank aggregation with application to grand challenges. / Fishbaugh, James; Prastawa, Marcel; Wang, Bo; Reynolds, Patrick; Aylward, Stephen; Gerig, Guido.

    Medical Image Computing and Computer Assisted Intervention − MICCAI 2017 - 20th International Conference, Proceedings. Vol. 10434 LNCS Springer Verlag, 2017. p. 754-762 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 10434 LNCS).

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Fishbaugh, J, Prastawa, M, Wang, B, Reynolds, P, Aylward, S & Gerig, G 2017, Data-driven rank aggregation with application to grand challenges. in Medical Image Computing and Computer Assisted Intervention − MICCAI 2017 - 20th International Conference, Proceedings. vol. 10434 LNCS, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 10434 LNCS, Springer Verlag, pp. 754-762, 20th International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2017, Quebec City, Canada, 9/11/17. https://doi.org/10.1007/978-3-319-66185-8_85
    Fishbaugh J, Prastawa M, Wang B, Reynolds P, Aylward S, Gerig G. Data-driven rank aggregation with application to grand challenges. In Medical Image Computing and Computer Assisted Intervention − MICCAI 2017 - 20th International Conference, Proceedings. Vol. 10434 LNCS. Springer Verlag. 2017. p. 754-762. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-319-66185-8_85
    Fishbaugh, James ; Prastawa, Marcel ; Wang, Bo ; Reynolds, Patrick ; Aylward, Stephen ; Gerig, Guido. / Data-driven rank aggregation with application to grand challenges. Medical Image Computing and Computer Assisted Intervention − MICCAI 2017 - 20th International Conference, Proceedings. Vol. 10434 LNCS Springer Verlag, 2017. pp. 754-762 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).
    @inproceedings{8371a2c960d8472f8868fc525b49ce56,
    title = "Data-driven rank aggregation with application to grand challenges",
    abstract = "The increased number of challenges for comparative evaluation of biomedical image analysis procedures clearly reflects a need for unbiased assessment of the state-of-the-art methodological advances. Moreover, the ultimate translation of novel image analysis procedures to the clinic requires rigorous validation and evaluation of alternative schemes, a task that is best outsourced to the international research community. We commonly see an increase of the number of metrics to be used in parallel, reflecting alternative ways to measure similarity. Since different measures come with different scales and distributions, these are often normalized or converted into an individual rank ordering, leaving the problem of combining the set of multiple rankings into a final score. Proposed solutions are averaging or accumulation of rankings, raising the question if different metrics are to be treated the same or if all metrics would be needed to assess closeness to truth. We address this issue with a data-driven method for automatic estimation of weights for a set of metrics based on unsupervised rank aggregation. Our method requires no normalization procedures and makes no assumptions about metric distributions. We explore the sensitivity of metrics to small changes in input data with an iterative perturbation scheme, to prioritize the contribution of the most robust metrics in the overall ranking. We show on real anatomical data that our weighting scheme can dramatically change the ranking.",
    author = "James Fishbaugh and Marcel Prastawa and Bo Wang and Patrick Reynolds and Stephen Aylward and Guido Gerig",
    year = "2017",
    doi = "10.1007/978-3-319-66185-8_85",
    language = "English (US)",
    isbn = "9783319661841",
    volume = "10434 LNCS",
    series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
    publisher = "Springer Verlag",
    pages = "754--762",
    booktitle = "Medical Image Computing and Computer Assisted Intervention − MICCAI 2017 - 20th International Conference, Proceedings",
    address = "Germany",

    }

    TY - GEN

    T1 - Data-driven rank aggregation with application to grand challenges

    AU - Fishbaugh, James

    AU - Prastawa, Marcel

    AU - Wang, Bo

    AU - Reynolds, Patrick

    AU - Aylward, Stephen

    AU - Gerig, Guido

    PY - 2017

    Y1 - 2017

    N2 - The increased number of challenges for comparative evaluation of biomedical image analysis procedures clearly reflects a need for unbiased assessment of the state-of-the-art methodological advances. Moreover, the ultimate translation of novel image analysis procedures to the clinic requires rigorous validation and evaluation of alternative schemes, a task that is best outsourced to the international research community. We commonly see an increase of the number of metrics to be used in parallel, reflecting alternative ways to measure similarity. Since different measures come with different scales and distributions, these are often normalized or converted into an individual rank ordering, leaving the problem of combining the set of multiple rankings into a final score. Proposed solutions are averaging or accumulation of rankings, raising the question if different metrics are to be treated the same or if all metrics would be needed to assess closeness to truth. We address this issue with a data-driven method for automatic estimation of weights for a set of metrics based on unsupervised rank aggregation. Our method requires no normalization procedures and makes no assumptions about metric distributions. We explore the sensitivity of metrics to small changes in input data with an iterative perturbation scheme, to prioritize the contribution of the most robust metrics in the overall ranking. We show on real anatomical data that our weighting scheme can dramatically change the ranking.

    AB - The increased number of challenges for comparative evaluation of biomedical image analysis procedures clearly reflects a need for unbiased assessment of the state-of-the-art methodological advances. Moreover, the ultimate translation of novel image analysis procedures to the clinic requires rigorous validation and evaluation of alternative schemes, a task that is best outsourced to the international research community. We commonly see an increase of the number of metrics to be used in parallel, reflecting alternative ways to measure similarity. Since different measures come with different scales and distributions, these are often normalized or converted into an individual rank ordering, leaving the problem of combining the set of multiple rankings into a final score. Proposed solutions are averaging or accumulation of rankings, raising the question if different metrics are to be treated the same or if all metrics would be needed to assess closeness to truth. We address this issue with a data-driven method for automatic estimation of weights for a set of metrics based on unsupervised rank aggregation. Our method requires no normalization procedures and makes no assumptions about metric distributions. We explore the sensitivity of metrics to small changes in input data with an iterative perturbation scheme, to prioritize the contribution of the most robust metrics in the overall ranking. We show on real anatomical data that our weighting scheme can dramatically change the ranking.

    UR - http://www.scopus.com/inward/record.url?scp=85029518644&partnerID=8YFLogxK

    UR - http://www.scopus.com/inward/citedby.url?scp=85029518644&partnerID=8YFLogxK

    U2 - 10.1007/978-3-319-66185-8_85

    DO - 10.1007/978-3-319-66185-8_85

    M3 - Conference contribution

    AN - SCOPUS:85029518644

    SN - 9783319661841

    VL - 10434 LNCS

    T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

    SP - 754

    EP - 762

    BT - Medical Image Computing and Computer Assisted Intervention − MICCAI 2017 - 20th International Conference, Proceedings

    PB - Springer Verlag

    ER -