Stability bounds for Non-i.i.d. processes

Mehryar Mohri, Afshin Rostamizadeh

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

The notion of algorithmic stability has been used effectively in the past to derive tight generalization bounds. A key advantage of these bounds is that they are designed for specific learning algorithms, exploiting their particular properties. But, as in much of learning theory, existing stability analyses and bounds apply only in the scenario where the samples are independently and identically distributed (i.i.d.). In many machine learning applications, however, this assumption does not hold. The observations received by the learning algorithm often have some inherent temporal dependence, which is clear in system diagnosis or time series prediction problems. This paper studies the scenario where the observations are drawn from a stationary mixing sequence, which implies a dependence between observations that weaken over time. It proves novel stability-based generalization bounds that hold even with this more general setting. These bounds strictly generalize the bounds given in the i.i.d. case. It also illustrates their application in the case of several general classes of learning algorithms, including Support Vector Regression and Kernel Ridge Regression.

Original languageEnglish (US)
Title of host publicationAdvances in Neural Information Processing Systems 20 - Proceedings of the 2007 Conference
StatePublished - 2009
Event21st Annual Conference on Neural Information Processing Systems, NIPS 2007 - Vancouver, BC, Canada
Duration: Dec 3 2007Dec 6 2007

Other

Other21st Annual Conference on Neural Information Processing Systems, NIPS 2007
CountryCanada
CityVancouver, BC
Period12/3/0712/6/07

Fingerprint

Learning algorithms
Learning systems
Time series

ASJC Scopus subject areas

  • Information Systems

Cite this

Mohri, M., & Rostamizadeh, A. (2009). Stability bounds for Non-i.i.d. processes. In Advances in Neural Information Processing Systems 20 - Proceedings of the 2007 Conference

Stability bounds for Non-i.i.d. processes. / Mohri, Mehryar; Rostamizadeh, Afshin.

Advances in Neural Information Processing Systems 20 - Proceedings of the 2007 Conference. 2009.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Mohri, M & Rostamizadeh, A 2009, Stability bounds for Non-i.i.d. processes. in Advances in Neural Information Processing Systems 20 - Proceedings of the 2007 Conference. 21st Annual Conference on Neural Information Processing Systems, NIPS 2007, Vancouver, BC, Canada, 12/3/07.
Mohri M, Rostamizadeh A. Stability bounds for Non-i.i.d. processes. In Advances in Neural Information Processing Systems 20 - Proceedings of the 2007 Conference. 2009
Mohri, Mehryar ; Rostamizadeh, Afshin. / Stability bounds for Non-i.i.d. processes. Advances in Neural Information Processing Systems 20 - Proceedings of the 2007 Conference. 2009.
@inproceedings{a20e50e82c67404bae32eba3c5beb631,
title = "Stability bounds for Non-i.i.d. processes",
abstract = "The notion of algorithmic stability has been used effectively in the past to derive tight generalization bounds. A key advantage of these bounds is that they are designed for specific learning algorithms, exploiting their particular properties. But, as in much of learning theory, existing stability analyses and bounds apply only in the scenario where the samples are independently and identically distributed (i.i.d.). In many machine learning applications, however, this assumption does not hold. The observations received by the learning algorithm often have some inherent temporal dependence, which is clear in system diagnosis or time series prediction problems. This paper studies the scenario where the observations are drawn from a stationary mixing sequence, which implies a dependence between observations that weaken over time. It proves novel stability-based generalization bounds that hold even with this more general setting. These bounds strictly generalize the bounds given in the i.i.d. case. It also illustrates their application in the case of several general classes of learning algorithms, including Support Vector Regression and Kernel Ridge Regression.",
author = "Mehryar Mohri and Afshin Rostamizadeh",
year = "2009",
language = "English (US)",
isbn = "160560352X",
booktitle = "Advances in Neural Information Processing Systems 20 - Proceedings of the 2007 Conference",

}

TY - GEN

T1 - Stability bounds for Non-i.i.d. processes

AU - Mohri, Mehryar

AU - Rostamizadeh, Afshin

PY - 2009

Y1 - 2009

N2 - The notion of algorithmic stability has been used effectively in the past to derive tight generalization bounds. A key advantage of these bounds is that they are designed for specific learning algorithms, exploiting their particular properties. But, as in much of learning theory, existing stability analyses and bounds apply only in the scenario where the samples are independently and identically distributed (i.i.d.). In many machine learning applications, however, this assumption does not hold. The observations received by the learning algorithm often have some inherent temporal dependence, which is clear in system diagnosis or time series prediction problems. This paper studies the scenario where the observations are drawn from a stationary mixing sequence, which implies a dependence between observations that weaken over time. It proves novel stability-based generalization bounds that hold even with this more general setting. These bounds strictly generalize the bounds given in the i.i.d. case. It also illustrates their application in the case of several general classes of learning algorithms, including Support Vector Regression and Kernel Ridge Regression.

AB - The notion of algorithmic stability has been used effectively in the past to derive tight generalization bounds. A key advantage of these bounds is that they are designed for specific learning algorithms, exploiting their particular properties. But, as in much of learning theory, existing stability analyses and bounds apply only in the scenario where the samples are independently and identically distributed (i.i.d.). In many machine learning applications, however, this assumption does not hold. The observations received by the learning algorithm often have some inherent temporal dependence, which is clear in system diagnosis or time series prediction problems. This paper studies the scenario where the observations are drawn from a stationary mixing sequence, which implies a dependence between observations that weaken over time. It proves novel stability-based generalization bounds that hold even with this more general setting. These bounds strictly generalize the bounds given in the i.i.d. case. It also illustrates their application in the case of several general classes of learning algorithms, including Support Vector Regression and Kernel Ridge Regression.

UR - http://www.scopus.com/inward/record.url?scp=84858785465&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84858785465&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:84858785465

SN - 160560352X

SN - 9781605603520

BT - Advances in Neural Information Processing Systems 20 - Proceedings of the 2007 Conference

ER -