Performance measures for classification systems with rejection

Filipe Condessa, José Bioucas-Dias, Jelena Kovacevic

Research output: Contribution to journalArticle

Abstract

Classifiers with rejection are essential in real-world applications where misclassifications and their effects are critical. However, if no problem specific cost function is defined, there are no established measures to assess the performance of such classifiers. We introduce a set of desired properties for performance measures for classifiers with rejection, based on which we propose a set of three performance measures for the evaluation of the performance of classifiers with rejection. The nonrejected accuracy measures the ability of the classifier to accurately classify nonrejected samples; the classification quality measures the correct decision making of the classifier with rejector; and the rejection quality measures the ability to concentrate all misclassified samples onto the set of rejected samples. We derive the concept of relative optimality that allows us to connect the measures to a family of cost functions that take into account the trade-off between rejection and misclassification. We illustrate the use of the proposed performance measures on classifiers with rejection applied to synthetic and real-world data.

Original languageEnglish (US)
Pages (from-to)437-450
Number of pages14
JournalPattern Recognition
Volume63
DOIs
StatePublished - Mar 1 2017

Fingerprint

Classifiers
Cost functions
Decision making

Keywords

  • Classification with rejection
  • Performance measures

ASJC Scopus subject areas

  • Software
  • Signal Processing
  • Computer Vision and Pattern Recognition
  • Artificial Intelligence

Cite this

Performance measures for classification systems with rejection. / Condessa, Filipe; Bioucas-Dias, José; Kovacevic, Jelena.

In: Pattern Recognition, Vol. 63, 01.03.2017, p. 437-450.

Research output: Contribution to journalArticle

Condessa, Filipe ; Bioucas-Dias, José ; Kovacevic, Jelena. / Performance measures for classification systems with rejection. In: Pattern Recognition. 2017 ; Vol. 63. pp. 437-450.
@article{495bc625354a42fcb2d75ecf8851fb65,
title = "Performance measures for classification systems with rejection",
abstract = "Classifiers with rejection are essential in real-world applications where misclassifications and their effects are critical. However, if no problem specific cost function is defined, there are no established measures to assess the performance of such classifiers. We introduce a set of desired properties for performance measures for classifiers with rejection, based on which we propose a set of three performance measures for the evaluation of the performance of classifiers with rejection. The nonrejected accuracy measures the ability of the classifier to accurately classify nonrejected samples; the classification quality measures the correct decision making of the classifier with rejector; and the rejection quality measures the ability to concentrate all misclassified samples onto the set of rejected samples. We derive the concept of relative optimality that allows us to connect the measures to a family of cost functions that take into account the trade-off between rejection and misclassification. We illustrate the use of the proposed performance measures on classifiers with rejection applied to synthetic and real-world data.",
keywords = "Classification with rejection, Performance measures",
author = "Filipe Condessa and Jos{\'e} Bioucas-Dias and Jelena Kovacevic",
year = "2017",
month = "3",
day = "1",
doi = "10.1016/j.patcog.2016.10.011",
language = "English (US)",
volume = "63",
pages = "437--450",
journal = "Pattern Recognition",
issn = "0031-3203",
publisher = "Elsevier Limited",

}

TY - JOUR

T1 - Performance measures for classification systems with rejection

AU - Condessa, Filipe

AU - Bioucas-Dias, José

AU - Kovacevic, Jelena

PY - 2017/3/1

Y1 - 2017/3/1

N2 - Classifiers with rejection are essential in real-world applications where misclassifications and their effects are critical. However, if no problem specific cost function is defined, there are no established measures to assess the performance of such classifiers. We introduce a set of desired properties for performance measures for classifiers with rejection, based on which we propose a set of three performance measures for the evaluation of the performance of classifiers with rejection. The nonrejected accuracy measures the ability of the classifier to accurately classify nonrejected samples; the classification quality measures the correct decision making of the classifier with rejector; and the rejection quality measures the ability to concentrate all misclassified samples onto the set of rejected samples. We derive the concept of relative optimality that allows us to connect the measures to a family of cost functions that take into account the trade-off between rejection and misclassification. We illustrate the use of the proposed performance measures on classifiers with rejection applied to synthetic and real-world data.

AB - Classifiers with rejection are essential in real-world applications where misclassifications and their effects are critical. However, if no problem specific cost function is defined, there are no established measures to assess the performance of such classifiers. We introduce a set of desired properties for performance measures for classifiers with rejection, based on which we propose a set of three performance measures for the evaluation of the performance of classifiers with rejection. The nonrejected accuracy measures the ability of the classifier to accurately classify nonrejected samples; the classification quality measures the correct decision making of the classifier with rejector; and the rejection quality measures the ability to concentrate all misclassified samples onto the set of rejected samples. We derive the concept of relative optimality that allows us to connect the measures to a family of cost functions that take into account the trade-off between rejection and misclassification. We illustrate the use of the proposed performance measures on classifiers with rejection applied to synthetic and real-world data.

KW - Classification with rejection

KW - Performance measures

UR - http://www.scopus.com/inward/record.url?scp=84999026115&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84999026115&partnerID=8YFLogxK

U2 - 10.1016/j.patcog.2016.10.011

DO - 10.1016/j.patcog.2016.10.011

M3 - Article

VL - 63

SP - 437

EP - 450

JO - Pattern Recognition

JF - Pattern Recognition

SN - 0031-3203

ER -