Domain adaptation and sample bias correction theory and algorithm for regression

Corinna Cortes, Mehryar Mohri

Research output: Contribution to journalArticle

Abstract

We present a series of new theoretical, algorithmic, and empirical results for domain adaptation and sample bias correction in regression. We prove that the discrepancy is a distance for the squared loss when the hypothesis set is the reproducing kernel Hilbert space induced by a universal kernel such as the Gaussian kernel. We give new pointwise loss guarantees based on the discrepancy of the empirical source and target distributions for the general class of kernel-based regularization algorithms. These bounds have a simpler form than previous results and hold for a broader class of convex loss functions not necessarily differentiable, including Lq losses and the hinge loss. We also give finer bounds based on the discrepancy and a weighted feature discrepancy parameter. We extend the discrepancy minimization adaptation algorithm to the more significant case where kernels are used and show that the problem can be cast as an SDP similar to the one in the feature space. We also show that techniques from smooth optimization can be used to derive an efficient algorithm for solving such SDPs even for very high-dimensional feature spaces and large samples. We have implemented this algorithm and report the results of experiments both with artificial and real-world data sets demonstrating its benefits both for general scenario of adaptation and the more specific scenario of sample bias correction. Our results show that it can scale to large data sets of tens of thousands or more points and demonstrate its performance improvement benefits.

Original languageEnglish (US)
Pages (from-to)103-126
Number of pages24
JournalTheoretical Computer Science
Volume519
DOIs
StatePublished - Jan 30 2014

Fingerprint

Bias Correction
Discrepancy
Regression
kernel
Feature Space
Scenarios
Gaussian Kernel
Reproducing Kernel Hilbert Space
Hilbert spaces
Hinges
Loss Function
Large Data Sets
Convex function
Differentiable
Regularization
High-dimensional
Efficient Algorithms
Target
Series
Optimization

Keywords

  • Domain adaptation
  • Learning theory
  • Machine learning
  • Optimization

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)

Cite this

Domain adaptation and sample bias correction theory and algorithm for regression. / Cortes, Corinna; Mohri, Mehryar.

In: Theoretical Computer Science, Vol. 519, 30.01.2014, p. 103-126.

Research output: Contribution to journalArticle

@article{2d073743dac6486e83ac9c03aef250f3,
title = "Domain adaptation and sample bias correction theory and algorithm for regression",
abstract = "We present a series of new theoretical, algorithmic, and empirical results for domain adaptation and sample bias correction in regression. We prove that the discrepancy is a distance for the squared loss when the hypothesis set is the reproducing kernel Hilbert space induced by a universal kernel such as the Gaussian kernel. We give new pointwise loss guarantees based on the discrepancy of the empirical source and target distributions for the general class of kernel-based regularization algorithms. These bounds have a simpler form than previous results and hold for a broader class of convex loss functions not necessarily differentiable, including Lq losses and the hinge loss. We also give finer bounds based on the discrepancy and a weighted feature discrepancy parameter. We extend the discrepancy minimization adaptation algorithm to the more significant case where kernels are used and show that the problem can be cast as an SDP similar to the one in the feature space. We also show that techniques from smooth optimization can be used to derive an efficient algorithm for solving such SDPs even for very high-dimensional feature spaces and large samples. We have implemented this algorithm and report the results of experiments both with artificial and real-world data sets demonstrating its benefits both for general scenario of adaptation and the more specific scenario of sample bias correction. Our results show that it can scale to large data sets of tens of thousands or more points and demonstrate its performance improvement benefits.",
keywords = "Domain adaptation, Learning theory, Machine learning, Optimization",
author = "Corinna Cortes and Mehryar Mohri",
year = "2014",
month = "1",
day = "30",
doi = "10.1016/j.tcs.2013.09.027",
language = "English (US)",
volume = "519",
pages = "103--126",
journal = "Theoretical Computer Science",
issn = "0304-3975",
publisher = "Elsevier",

}

TY - JOUR

T1 - Domain adaptation and sample bias correction theory and algorithm for regression

AU - Cortes, Corinna

AU - Mohri, Mehryar

PY - 2014/1/30

Y1 - 2014/1/30

N2 - We present a series of new theoretical, algorithmic, and empirical results for domain adaptation and sample bias correction in regression. We prove that the discrepancy is a distance for the squared loss when the hypothesis set is the reproducing kernel Hilbert space induced by a universal kernel such as the Gaussian kernel. We give new pointwise loss guarantees based on the discrepancy of the empirical source and target distributions for the general class of kernel-based regularization algorithms. These bounds have a simpler form than previous results and hold for a broader class of convex loss functions not necessarily differentiable, including Lq losses and the hinge loss. We also give finer bounds based on the discrepancy and a weighted feature discrepancy parameter. We extend the discrepancy minimization adaptation algorithm to the more significant case where kernels are used and show that the problem can be cast as an SDP similar to the one in the feature space. We also show that techniques from smooth optimization can be used to derive an efficient algorithm for solving such SDPs even for very high-dimensional feature spaces and large samples. We have implemented this algorithm and report the results of experiments both with artificial and real-world data sets demonstrating its benefits both for general scenario of adaptation and the more specific scenario of sample bias correction. Our results show that it can scale to large data sets of tens of thousands or more points and demonstrate its performance improvement benefits.

AB - We present a series of new theoretical, algorithmic, and empirical results for domain adaptation and sample bias correction in regression. We prove that the discrepancy is a distance for the squared loss when the hypothesis set is the reproducing kernel Hilbert space induced by a universal kernel such as the Gaussian kernel. We give new pointwise loss guarantees based on the discrepancy of the empirical source and target distributions for the general class of kernel-based regularization algorithms. These bounds have a simpler form than previous results and hold for a broader class of convex loss functions not necessarily differentiable, including Lq losses and the hinge loss. We also give finer bounds based on the discrepancy and a weighted feature discrepancy parameter. We extend the discrepancy minimization adaptation algorithm to the more significant case where kernels are used and show that the problem can be cast as an SDP similar to the one in the feature space. We also show that techniques from smooth optimization can be used to derive an efficient algorithm for solving such SDPs even for very high-dimensional feature spaces and large samples. We have implemented this algorithm and report the results of experiments both with artificial and real-world data sets demonstrating its benefits both for general scenario of adaptation and the more specific scenario of sample bias correction. Our results show that it can scale to large data sets of tens of thousands or more points and demonstrate its performance improvement benefits.

KW - Domain adaptation

KW - Learning theory

KW - Machine learning

KW - Optimization

UR - http://www.scopus.com/inward/record.url?scp=84892371351&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84892371351&partnerID=8YFLogxK

U2 - 10.1016/j.tcs.2013.09.027

DO - 10.1016/j.tcs.2013.09.027

M3 - Article

VL - 519

SP - 103

EP - 126

JO - Theoretical Computer Science

JF - Theoretical Computer Science

SN - 0304-3975

ER -