Confidence intervals for the area under the ROC Curve

Corinna Cortes, Mehryar Mohri

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

In many applications, good ranking is a highly desirable performance for a classifier. The criterion commonly used to measure the ranking quality of a classification algorithm is the area under the ROC curve (AUC). To report it properly, it is crucial to determine an interval of confidence for its value. This paper provides confidence intervals for the AUC based on a statistical and combinatorial analysis using only simple parameters such as the error rate and the number of positive and negative examples. The analysis is distribution-independent, it makes no assumption about the distribution of the scores of negative or positive examples. The results are of practical use and can be viewed as the equivalent for AUC of the standard confidence intervals given in the case of the error rate. They are compared with previous approaches in several standard classification tasks demonstrating the benefits of our analysis.

Original languageEnglish (US)
Title of host publicationAdvances in Neural Information Processing Systems 17 - Proceedings of the 2004 Conference, NIPS 2004
PublisherNeural information processing systems foundation
ISBN (Print)0262195348, 9780262195348
StatePublished - 2005
Event18th Annual Conference on Neural Information Processing Systems, NIPS 2004 - Vancouver, BC, Canada
Duration: Dec 13 2004Dec 16 2004

Other

Other18th Annual Conference on Neural Information Processing Systems, NIPS 2004
CountryCanada
CityVancouver, BC
Period12/13/0412/16/04

Fingerprint

Classifiers

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Information Systems
  • Signal Processing

Cite this

Cortes, C., & Mohri, M. (2005). Confidence intervals for the area under the ROC Curve. In Advances in Neural Information Processing Systems 17 - Proceedings of the 2004 Conference, NIPS 2004 Neural information processing systems foundation.

Confidence intervals for the area under the ROC Curve. / Cortes, Corinna; Mohri, Mehryar.

Advances in Neural Information Processing Systems 17 - Proceedings of the 2004 Conference, NIPS 2004. Neural information processing systems foundation, 2005.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Cortes, C & Mohri, M 2005, Confidence intervals for the area under the ROC Curve. in Advances in Neural Information Processing Systems 17 - Proceedings of the 2004 Conference, NIPS 2004. Neural information processing systems foundation, 18th Annual Conference on Neural Information Processing Systems, NIPS 2004, Vancouver, BC, Canada, 12/13/04.
Cortes C, Mohri M. Confidence intervals for the area under the ROC Curve. In Advances in Neural Information Processing Systems 17 - Proceedings of the 2004 Conference, NIPS 2004. Neural information processing systems foundation. 2005
Cortes, Corinna ; Mohri, Mehryar. / Confidence intervals for the area under the ROC Curve. Advances in Neural Information Processing Systems 17 - Proceedings of the 2004 Conference, NIPS 2004. Neural information processing systems foundation, 2005.
@inproceedings{5117d649893943ae8e9a72c527eadb29,
title = "Confidence intervals for the area under the ROC Curve",
abstract = "In many applications, good ranking is a highly desirable performance for a classifier. The criterion commonly used to measure the ranking quality of a classification algorithm is the area under the ROC curve (AUC). To report it properly, it is crucial to determine an interval of confidence for its value. This paper provides confidence intervals for the AUC based on a statistical and combinatorial analysis using only simple parameters such as the error rate and the number of positive and negative examples. The analysis is distribution-independent, it makes no assumption about the distribution of the scores of negative or positive examples. The results are of practical use and can be viewed as the equivalent for AUC of the standard confidence intervals given in the case of the error rate. They are compared with previous approaches in several standard classification tasks demonstrating the benefits of our analysis.",
author = "Corinna Cortes and Mehryar Mohri",
year = "2005",
language = "English (US)",
isbn = "0262195348",
booktitle = "Advances in Neural Information Processing Systems 17 - Proceedings of the 2004 Conference, NIPS 2004",
publisher = "Neural information processing systems foundation",

}

TY - GEN

T1 - Confidence intervals for the area under the ROC Curve

AU - Cortes, Corinna

AU - Mohri, Mehryar

PY - 2005

Y1 - 2005

N2 - In many applications, good ranking is a highly desirable performance for a classifier. The criterion commonly used to measure the ranking quality of a classification algorithm is the area under the ROC curve (AUC). To report it properly, it is crucial to determine an interval of confidence for its value. This paper provides confidence intervals for the AUC based on a statistical and combinatorial analysis using only simple parameters such as the error rate and the number of positive and negative examples. The analysis is distribution-independent, it makes no assumption about the distribution of the scores of negative or positive examples. The results are of practical use and can be viewed as the equivalent for AUC of the standard confidence intervals given in the case of the error rate. They are compared with previous approaches in several standard classification tasks demonstrating the benefits of our analysis.

AB - In many applications, good ranking is a highly desirable performance for a classifier. The criterion commonly used to measure the ranking quality of a classification algorithm is the area under the ROC curve (AUC). To report it properly, it is crucial to determine an interval of confidence for its value. This paper provides confidence intervals for the AUC based on a statistical and combinatorial analysis using only simple parameters such as the error rate and the number of positive and negative examples. The analysis is distribution-independent, it makes no assumption about the distribution of the scores of negative or positive examples. The results are of practical use and can be viewed as the equivalent for AUC of the standard confidence intervals given in the case of the error rate. They are compared with previous approaches in several standard classification tasks demonstrating the benefits of our analysis.

UR - http://www.scopus.com/inward/record.url?scp=84898972495&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84898972495&partnerID=8YFLogxK

M3 - Conference contribution

SN - 0262195348

SN - 9780262195348

BT - Advances in Neural Information Processing Systems 17 - Proceedings of the 2004 Conference, NIPS 2004

PB - Neural information processing systems foundation

ER -