Asymptotics of MAP Inference in Deep Networks

Parthe Pandit, Mojtaba Sahraee, Sundeep Rangan, Alyson K. Fletcher

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Deep generative priors are a powerful tool for reconstruction problems with complex data such as images and text. Inverse problems using such models require solving an inference problem of estimating the input and hidden units of the multi-layer network from its output. Maximum a priori (MAP) estimation is a widely-used inference method as it is straightforward to implement, and has been successful in practice. However, rigorous analysis of MAP inference in multi-layer networks is difficult. This work considers a recently-developed method, multilayer vector approximate message passing (ML-VAMP), to study MAP inference in deep networks. It is shown that the mean squared error of the ML-VAMP estimate can be exactly and rigorously characterized in a certain high-dimensional random limit. The proposed method thus provides a tractable method for MAP inference with exact performance guarantees.

Original languageEnglish (US)
Title of host publication2019 IEEE International Symposium on Information Theory, ISIT 2019 - Proceedings
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages842-846
Number of pages5
ISBN (Electronic)9781538692912
DOIs
StatePublished - Jul 2019
Event2019 IEEE International Symposium on Information Theory, ISIT 2019 - Paris, France
Duration: Jul 7 2019Jul 12 2019

Publication series

NameIEEE International Symposium on Information Theory - Proceedings
Volume2019-July
ISSN (Print)2157-8095

Conference

Conference2019 IEEE International Symposium on Information Theory, ISIT 2019
CountryFrance
CityParis
Period7/7/197/12/19

Fingerprint

Network layers
Message passing
Multilayers
Multilayer
Inverse problems
Message Passing
Performance Guarantee
Mean Squared Error
Inverse Problem
High-dimensional
Unit
Output
Estimate

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Information Systems
  • Modeling and Simulation
  • Applied Mathematics

Cite this

Pandit, P., Sahraee, M., Rangan, S., & Fletcher, A. K. (2019). Asymptotics of MAP Inference in Deep Networks. In 2019 IEEE International Symposium on Information Theory, ISIT 2019 - Proceedings (pp. 842-846). [8849316] (IEEE International Symposium on Information Theory - Proceedings; Vol. 2019-July). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ISIT.2019.8849316

Asymptotics of MAP Inference in Deep Networks. / Pandit, Parthe; Sahraee, Mojtaba; Rangan, Sundeep; Fletcher, Alyson K.

2019 IEEE International Symposium on Information Theory, ISIT 2019 - Proceedings. Institute of Electrical and Electronics Engineers Inc., 2019. p. 842-846 8849316 (IEEE International Symposium on Information Theory - Proceedings; Vol. 2019-July).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Pandit, P, Sahraee, M, Rangan, S & Fletcher, AK 2019, Asymptotics of MAP Inference in Deep Networks. in 2019 IEEE International Symposium on Information Theory, ISIT 2019 - Proceedings., 8849316, IEEE International Symposium on Information Theory - Proceedings, vol. 2019-July, Institute of Electrical and Electronics Engineers Inc., pp. 842-846, 2019 IEEE International Symposium on Information Theory, ISIT 2019, Paris, France, 7/7/19. https://doi.org/10.1109/ISIT.2019.8849316
Pandit P, Sahraee M, Rangan S, Fletcher AK. Asymptotics of MAP Inference in Deep Networks. In 2019 IEEE International Symposium on Information Theory, ISIT 2019 - Proceedings. Institute of Electrical and Electronics Engineers Inc. 2019. p. 842-846. 8849316. (IEEE International Symposium on Information Theory - Proceedings). https://doi.org/10.1109/ISIT.2019.8849316
Pandit, Parthe ; Sahraee, Mojtaba ; Rangan, Sundeep ; Fletcher, Alyson K. / Asymptotics of MAP Inference in Deep Networks. 2019 IEEE International Symposium on Information Theory, ISIT 2019 - Proceedings. Institute of Electrical and Electronics Engineers Inc., 2019. pp. 842-846 (IEEE International Symposium on Information Theory - Proceedings).
@inproceedings{5c5a6494a6814d25b0c7f0b1e83f57bd,
title = "Asymptotics of MAP Inference in Deep Networks",
abstract = "Deep generative priors are a powerful tool for reconstruction problems with complex data such as images and text. Inverse problems using such models require solving an inference problem of estimating the input and hidden units of the multi-layer network from its output. Maximum a priori (MAP) estimation is a widely-used inference method as it is straightforward to implement, and has been successful in practice. However, rigorous analysis of MAP inference in multi-layer networks is difficult. This work considers a recently-developed method, multilayer vector approximate message passing (ML-VAMP), to study MAP inference in deep networks. It is shown that the mean squared error of the ML-VAMP estimate can be exactly and rigorously characterized in a certain high-dimensional random limit. The proposed method thus provides a tractable method for MAP inference with exact performance guarantees.",
author = "Parthe Pandit and Mojtaba Sahraee and Sundeep Rangan and Fletcher, {Alyson K.}",
year = "2019",
month = "7",
doi = "10.1109/ISIT.2019.8849316",
language = "English (US)",
series = "IEEE International Symposium on Information Theory - Proceedings",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
pages = "842--846",
booktitle = "2019 IEEE International Symposium on Information Theory, ISIT 2019 - Proceedings",

}

TY - GEN

T1 - Asymptotics of MAP Inference in Deep Networks

AU - Pandit, Parthe

AU - Sahraee, Mojtaba

AU - Rangan, Sundeep

AU - Fletcher, Alyson K.

PY - 2019/7

Y1 - 2019/7

N2 - Deep generative priors are a powerful tool for reconstruction problems with complex data such as images and text. Inverse problems using such models require solving an inference problem of estimating the input and hidden units of the multi-layer network from its output. Maximum a priori (MAP) estimation is a widely-used inference method as it is straightforward to implement, and has been successful in practice. However, rigorous analysis of MAP inference in multi-layer networks is difficult. This work considers a recently-developed method, multilayer vector approximate message passing (ML-VAMP), to study MAP inference in deep networks. It is shown that the mean squared error of the ML-VAMP estimate can be exactly and rigorously characterized in a certain high-dimensional random limit. The proposed method thus provides a tractable method for MAP inference with exact performance guarantees.

AB - Deep generative priors are a powerful tool for reconstruction problems with complex data such as images and text. Inverse problems using such models require solving an inference problem of estimating the input and hidden units of the multi-layer network from its output. Maximum a priori (MAP) estimation is a widely-used inference method as it is straightforward to implement, and has been successful in practice. However, rigorous analysis of MAP inference in multi-layer networks is difficult. This work considers a recently-developed method, multilayer vector approximate message passing (ML-VAMP), to study MAP inference in deep networks. It is shown that the mean squared error of the ML-VAMP estimate can be exactly and rigorously characterized in a certain high-dimensional random limit. The proposed method thus provides a tractable method for MAP inference with exact performance guarantees.

UR - http://www.scopus.com/inward/record.url?scp=85073171878&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85073171878&partnerID=8YFLogxK

U2 - 10.1109/ISIT.2019.8849316

DO - 10.1109/ISIT.2019.8849316

M3 - Conference contribution

AN - SCOPUS:85073171878

T3 - IEEE International Symposium on Information Theory - Proceedings

SP - 842

EP - 846

BT - 2019 IEEE International Symposium on Information Theory, ISIT 2019 - Proceedings

PB - Institute of Electrical and Electronics Engineers Inc.

ER -