Multi-class deep boosting

Vitaly Kuznetsov, Mehryar Mohri, Umar Syed

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

We present new ensemble learning algorithms for multi-class classification. Our algorithms can use as a base classifier set a family of deep decision trees or other rich or complex families and yet benefit from strong generalization guarantees. We give new data-dependent learning bounds for convex ensembles in the multi-class classification setting expressed in terms of the Rademacher complexities of the sub-families composing the base classifier set, and the mixture weight assigned to each sub-family. These bounds are finer than existing ones both thanks to an improved dependency on the number of classes and, more crucially, by virtue of a more favorable complexity term expressed as an average of the Rademacher complexities based on the ensemble's mixture weights. We introduce and discuss several new multi-class ensemble algorithms benefiting from these guarantees, prove positive results for the H-consistency of several of them, and report the results of experiments showing that their performance compares favorably with that of multi-class versions of AdaBoost and Logistic Regression and their L<inf>1</inf>-regularized counterparts.

Original languageEnglish (US)
Title of host publicationAdvances in Neural Information Processing Systems
PublisherNeural information processing systems foundation
Pages2501-2509
Number of pages9
Volume3
EditionJanuary
StatePublished - 2014
Event28th Annual Conference on Neural Information Processing Systems 2014, NIPS 2014 - Montreal, Canada
Duration: Dec 8 2014Dec 13 2014

Other

Other28th Annual Conference on Neural Information Processing Systems 2014, NIPS 2014
CountryCanada
CityMontreal
Period12/8/1412/13/14

Fingerprint

Classifiers
Adaptive boosting
Decision trees
Learning algorithms
Logistics
Experiments

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Information Systems
  • Signal Processing

Cite this

Kuznetsov, V., Mohri, M., & Syed, U. (2014). Multi-class deep boosting. In Advances in Neural Information Processing Systems (January ed., Vol. 3, pp. 2501-2509). Neural information processing systems foundation.

Multi-class deep boosting. / Kuznetsov, Vitaly; Mohri, Mehryar; Syed, Umar.

Advances in Neural Information Processing Systems. Vol. 3 January. ed. Neural information processing systems foundation, 2014. p. 2501-2509.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Kuznetsov, V, Mohri, M & Syed, U 2014, Multi-class deep boosting. in Advances in Neural Information Processing Systems. January edn, vol. 3, Neural information processing systems foundation, pp. 2501-2509, 28th Annual Conference on Neural Information Processing Systems 2014, NIPS 2014, Montreal, Canada, 12/8/14.
Kuznetsov V, Mohri M, Syed U. Multi-class deep boosting. In Advances in Neural Information Processing Systems. January ed. Vol. 3. Neural information processing systems foundation. 2014. p. 2501-2509
Kuznetsov, Vitaly ; Mohri, Mehryar ; Syed, Umar. / Multi-class deep boosting. Advances in Neural Information Processing Systems. Vol. 3 January. ed. Neural information processing systems foundation, 2014. pp. 2501-2509
@inproceedings{398e558e86be4ada86fc8ec70fad01f4,
title = "Multi-class deep boosting",
abstract = "We present new ensemble learning algorithms for multi-class classification. Our algorithms can use as a base classifier set a family of deep decision trees or other rich or complex families and yet benefit from strong generalization guarantees. We give new data-dependent learning bounds for convex ensembles in the multi-class classification setting expressed in terms of the Rademacher complexities of the sub-families composing the base classifier set, and the mixture weight assigned to each sub-family. These bounds are finer than existing ones both thanks to an improved dependency on the number of classes and, more crucially, by virtue of a more favorable complexity term expressed as an average of the Rademacher complexities based on the ensemble's mixture weights. We introduce and discuss several new multi-class ensemble algorithms benefiting from these guarantees, prove positive results for the H-consistency of several of them, and report the results of experiments showing that their performance compares favorably with that of multi-class versions of AdaBoost and Logistic Regression and their L1-regularized counterparts.",
author = "Vitaly Kuznetsov and Mehryar Mohri and Umar Syed",
year = "2014",
language = "English (US)",
volume = "3",
pages = "2501--2509",
booktitle = "Advances in Neural Information Processing Systems",
publisher = "Neural information processing systems foundation",
edition = "January",

}

TY - GEN

T1 - Multi-class deep boosting

AU - Kuznetsov, Vitaly

AU - Mohri, Mehryar

AU - Syed, Umar

PY - 2014

Y1 - 2014

N2 - We present new ensemble learning algorithms for multi-class classification. Our algorithms can use as a base classifier set a family of deep decision trees or other rich or complex families and yet benefit from strong generalization guarantees. We give new data-dependent learning bounds for convex ensembles in the multi-class classification setting expressed in terms of the Rademacher complexities of the sub-families composing the base classifier set, and the mixture weight assigned to each sub-family. These bounds are finer than existing ones both thanks to an improved dependency on the number of classes and, more crucially, by virtue of a more favorable complexity term expressed as an average of the Rademacher complexities based on the ensemble's mixture weights. We introduce and discuss several new multi-class ensemble algorithms benefiting from these guarantees, prove positive results for the H-consistency of several of them, and report the results of experiments showing that their performance compares favorably with that of multi-class versions of AdaBoost and Logistic Regression and their L1-regularized counterparts.

AB - We present new ensemble learning algorithms for multi-class classification. Our algorithms can use as a base classifier set a family of deep decision trees or other rich or complex families and yet benefit from strong generalization guarantees. We give new data-dependent learning bounds for convex ensembles in the multi-class classification setting expressed in terms of the Rademacher complexities of the sub-families composing the base classifier set, and the mixture weight assigned to each sub-family. These bounds are finer than existing ones both thanks to an improved dependency on the number of classes and, more crucially, by virtue of a more favorable complexity term expressed as an average of the Rademacher complexities based on the ensemble's mixture weights. We introduce and discuss several new multi-class ensemble algorithms benefiting from these guarantees, prove positive results for the H-consistency of several of them, and report the results of experiments showing that their performance compares favorably with that of multi-class versions of AdaBoost and Logistic Regression and their L1-regularized counterparts.

UR - http://www.scopus.com/inward/record.url?scp=84937956673&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84937956673&partnerID=8YFLogxK

M3 - Conference contribution

VL - 3

SP - 2501

EP - 2509

BT - Advances in Neural Information Processing Systems

PB - Neural information processing systems foundation

ER -