Structural maxent models

Corinna Cortes, Vitaly Kuznetsov, Mehryar Mohri, Umar Syed

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

We present a new class of density estimation models, Structural Maxent models, with feature functions selected from a union of possibly very complex sub-families and yet benefiting from strong learning guarantees. The design of our models is based on a new principle supported by uniform convergence bounds and taking into consideration the complexity of the different sub-families composing the full set of features. We prove new data-dependent learning bounds for our models, expressed in terms of the Rademacher complexities of these sub-families. We also prove a duality theorem, which we use to derive our Structural Maxent algorithm. We give a full description of our algorithm, including the details of its derivation, and report the results of several experiments demonstrating that its performance improves on that of existing Li-norm regularized Maxent algorithms. We further similarly define conditional Structural Maxent models for multi-class classification problems. These are conditional probability models also making use of a union of possibly complex feature subfamilies. We prove a duality theorem for these models as well, which reveals their connection with existing binary and multi-class deep boosting algorithms.

Original languageEnglish (US)
Title of host publication32nd International Conference on Machine Learning, ICML 2015
PublisherInternational Machine Learning Society (IMLS)
Pages391-399
Number of pages9
Volume1
ISBN (Print)9781510810587
StatePublished - 2015
Event32nd International Conference on Machine Learning, ICML 2015 - Lile, France
Duration: Jul 6 2015Jul 11 2015

Other

Other32nd International Conference on Machine Learning, ICML 2015
CountryFrance
CityLile
Period7/6/157/11/15

Fingerprint

Experiments

ASJC Scopus subject areas

  • Human-Computer Interaction
  • Computer Science Applications

Cite this

Cortes, C., Kuznetsov, V., Mohri, M., & Syed, U. (2015). Structural maxent models. In 32nd International Conference on Machine Learning, ICML 2015 (Vol. 1, pp. 391-399). International Machine Learning Society (IMLS).

Structural maxent models. / Cortes, Corinna; Kuznetsov, Vitaly; Mohri, Mehryar; Syed, Umar.

32nd International Conference on Machine Learning, ICML 2015. Vol. 1 International Machine Learning Society (IMLS), 2015. p. 391-399.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Cortes, C, Kuznetsov, V, Mohri, M & Syed, U 2015, Structural maxent models. in 32nd International Conference on Machine Learning, ICML 2015. vol. 1, International Machine Learning Society (IMLS), pp. 391-399, 32nd International Conference on Machine Learning, ICML 2015, Lile, France, 7/6/15.
Cortes C, Kuznetsov V, Mohri M, Syed U. Structural maxent models. In 32nd International Conference on Machine Learning, ICML 2015. Vol. 1. International Machine Learning Society (IMLS). 2015. p. 391-399
Cortes, Corinna ; Kuznetsov, Vitaly ; Mohri, Mehryar ; Syed, Umar. / Structural maxent models. 32nd International Conference on Machine Learning, ICML 2015. Vol. 1 International Machine Learning Society (IMLS), 2015. pp. 391-399
@inproceedings{badf568e92a147a791194a570ff8228a,
title = "Structural maxent models",
abstract = "We present a new class of density estimation models, Structural Maxent models, with feature functions selected from a union of possibly very complex sub-families and yet benefiting from strong learning guarantees. The design of our models is based on a new principle supported by uniform convergence bounds and taking into consideration the complexity of the different sub-families composing the full set of features. We prove new data-dependent learning bounds for our models, expressed in terms of the Rademacher complexities of these sub-families. We also prove a duality theorem, which we use to derive our Structural Maxent algorithm. We give a full description of our algorithm, including the details of its derivation, and report the results of several experiments demonstrating that its performance improves on that of existing Li-norm regularized Maxent algorithms. We further similarly define conditional Structural Maxent models for multi-class classification problems. These are conditional probability models also making use of a union of possibly complex feature subfamilies. We prove a duality theorem for these models as well, which reveals their connection with existing binary and multi-class deep boosting algorithms.",
author = "Corinna Cortes and Vitaly Kuznetsov and Mehryar Mohri and Umar Syed",
year = "2015",
language = "English (US)",
isbn = "9781510810587",
volume = "1",
pages = "391--399",
booktitle = "32nd International Conference on Machine Learning, ICML 2015",
publisher = "International Machine Learning Society (IMLS)",

}

TY - GEN

T1 - Structural maxent models

AU - Cortes, Corinna

AU - Kuznetsov, Vitaly

AU - Mohri, Mehryar

AU - Syed, Umar

PY - 2015

Y1 - 2015

N2 - We present a new class of density estimation models, Structural Maxent models, with feature functions selected from a union of possibly very complex sub-families and yet benefiting from strong learning guarantees. The design of our models is based on a new principle supported by uniform convergence bounds and taking into consideration the complexity of the different sub-families composing the full set of features. We prove new data-dependent learning bounds for our models, expressed in terms of the Rademacher complexities of these sub-families. We also prove a duality theorem, which we use to derive our Structural Maxent algorithm. We give a full description of our algorithm, including the details of its derivation, and report the results of several experiments demonstrating that its performance improves on that of existing Li-norm regularized Maxent algorithms. We further similarly define conditional Structural Maxent models for multi-class classification problems. These are conditional probability models also making use of a union of possibly complex feature subfamilies. We prove a duality theorem for these models as well, which reveals their connection with existing binary and multi-class deep boosting algorithms.

AB - We present a new class of density estimation models, Structural Maxent models, with feature functions selected from a union of possibly very complex sub-families and yet benefiting from strong learning guarantees. The design of our models is based on a new principle supported by uniform convergence bounds and taking into consideration the complexity of the different sub-families composing the full set of features. We prove new data-dependent learning bounds for our models, expressed in terms of the Rademacher complexities of these sub-families. We also prove a duality theorem, which we use to derive our Structural Maxent algorithm. We give a full description of our algorithm, including the details of its derivation, and report the results of several experiments demonstrating that its performance improves on that of existing Li-norm regularized Maxent algorithms. We further similarly define conditional Structural Maxent models for multi-class classification problems. These are conditional probability models also making use of a union of possibly complex feature subfamilies. We prove a duality theorem for these models as well, which reveals their connection with existing binary and multi-class deep boosting algorithms.

UR - http://www.scopus.com/inward/record.url?scp=84969504843&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84969504843&partnerID=8YFLogxK

M3 - Conference contribution

SN - 9781510810587

VL - 1

SP - 391

EP - 399

BT - 32nd International Conference on Machine Learning, ICML 2015

PB - International Machine Learning Society (IMLS)

ER -