Conjugate conformal prediction for online binary classification

Mustafa A. Kocak, Elza Erkip, Dennis Shasha

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Binary classification (rain or shine, disease or not, increase or decrease) is a fundamental problem in machine learning. We present an algorithm that can take any standard online binary classification algorithm and provably improve its performance under very weak assumptions, given the right to refuse to make predictions in certain cases. The extent of improvement will depend on the data size, stability of the algorithm, and room for improvement in the algorithms performance. Our experiments on standard machine learning data sets and standard algorithms (k-nearest neighbors and random forests) show the effectiveness of our approach, even beyond what is possible using previous work on conformal predictors upon which our approach is based. Though we focus on binary classification, our theory could be extended to multiway classification. Our code and data are available upon request.

Original languageEnglish (US)
Title of host publication32nd Conference on Uncertainty in Artificial Intelligence 2016, UAI 2016
PublisherAssociation For Uncertainty in Artificial Intelligence (AUAI)
Pages347-356
Number of pages10
ISBN (Electronic)9781510827806
StatePublished - 2016
Event32nd Conference on Uncertainty in Artificial Intelligence 2016, UAI 2016 - Jersey City, United States
Duration: Jun 25 2016Jun 29 2016

Other

Other32nd Conference on Uncertainty in Artificial Intelligence 2016, UAI 2016
CountryUnited States
CityJersey City
Period6/25/166/29/16

Fingerprint

Learning systems
Rain
Experiments

ASJC Scopus subject areas

  • Artificial Intelligence

Cite this

Kocak, M. A., Erkip, E., & Shasha, D. (2016). Conjugate conformal prediction for online binary classification. In 32nd Conference on Uncertainty in Artificial Intelligence 2016, UAI 2016 (pp. 347-356). Association For Uncertainty in Artificial Intelligence (AUAI).

Conjugate conformal prediction for online binary classification. / Kocak, Mustafa A.; Erkip, Elza; Shasha, Dennis.

32nd Conference on Uncertainty in Artificial Intelligence 2016, UAI 2016. Association For Uncertainty in Artificial Intelligence (AUAI), 2016. p. 347-356.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Kocak, MA, Erkip, E & Shasha, D 2016, Conjugate conformal prediction for online binary classification. in 32nd Conference on Uncertainty in Artificial Intelligence 2016, UAI 2016. Association For Uncertainty in Artificial Intelligence (AUAI), pp. 347-356, 32nd Conference on Uncertainty in Artificial Intelligence 2016, UAI 2016, Jersey City, United States, 6/25/16.
Kocak MA, Erkip E, Shasha D. Conjugate conformal prediction for online binary classification. In 32nd Conference on Uncertainty in Artificial Intelligence 2016, UAI 2016. Association For Uncertainty in Artificial Intelligence (AUAI). 2016. p. 347-356
Kocak, Mustafa A. ; Erkip, Elza ; Shasha, Dennis. / Conjugate conformal prediction for online binary classification. 32nd Conference on Uncertainty in Artificial Intelligence 2016, UAI 2016. Association For Uncertainty in Artificial Intelligence (AUAI), 2016. pp. 347-356
@inproceedings{b1858127bcc443b588ccf71c865347d5,
title = "Conjugate conformal prediction for online binary classification",
abstract = "Binary classification (rain or shine, disease or not, increase or decrease) is a fundamental problem in machine learning. We present an algorithm that can take any standard online binary classification algorithm and provably improve its performance under very weak assumptions, given the right to refuse to make predictions in certain cases. The extent of improvement will depend on the data size, stability of the algorithm, and room for improvement in the algorithms performance. Our experiments on standard machine learning data sets and standard algorithms (k-nearest neighbors and random forests) show the effectiveness of our approach, even beyond what is possible using previous work on conformal predictors upon which our approach is based. Though we focus on binary classification, our theory could be extended to multiway classification. Our code and data are available upon request.",
author = "Kocak, {Mustafa A.} and Elza Erkip and Dennis Shasha",
year = "2016",
language = "English (US)",
pages = "347--356",
booktitle = "32nd Conference on Uncertainty in Artificial Intelligence 2016, UAI 2016",
publisher = "Association For Uncertainty in Artificial Intelligence (AUAI)",

}

TY - GEN

T1 - Conjugate conformal prediction for online binary classification

AU - Kocak, Mustafa A.

AU - Erkip, Elza

AU - Shasha, Dennis

PY - 2016

Y1 - 2016

N2 - Binary classification (rain or shine, disease or not, increase or decrease) is a fundamental problem in machine learning. We present an algorithm that can take any standard online binary classification algorithm and provably improve its performance under very weak assumptions, given the right to refuse to make predictions in certain cases. The extent of improvement will depend on the data size, stability of the algorithm, and room for improvement in the algorithms performance. Our experiments on standard machine learning data sets and standard algorithms (k-nearest neighbors and random forests) show the effectiveness of our approach, even beyond what is possible using previous work on conformal predictors upon which our approach is based. Though we focus on binary classification, our theory could be extended to multiway classification. Our code and data are available upon request.

AB - Binary classification (rain or shine, disease or not, increase or decrease) is a fundamental problem in machine learning. We present an algorithm that can take any standard online binary classification algorithm and provably improve its performance under very weak assumptions, given the right to refuse to make predictions in certain cases. The extent of improvement will depend on the data size, stability of the algorithm, and room for improvement in the algorithms performance. Our experiments on standard machine learning data sets and standard algorithms (k-nearest neighbors and random forests) show the effectiveness of our approach, even beyond what is possible using previous work on conformal predictors upon which our approach is based. Though we focus on binary classification, our theory could be extended to multiway classification. Our code and data are available upon request.

UR - http://www.scopus.com/inward/record.url?scp=85001967813&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85001967813&partnerID=8YFLogxK

M3 - Conference contribution

SP - 347

EP - 356

BT - 32nd Conference on Uncertainty in Artificial Intelligence 2016, UAI 2016

PB - Association For Uncertainty in Artificial Intelligence (AUAI)

ER -