Binary embeddings with structured hashed projections

Anna Choromanska, Krzysztof Choromanski, Mariusz Bojarski, Tony Jebara, Sanjiv Kumar, Yann LeCun

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

We consider the hashing mechanism for constructing binary embeddings, that involves pseudo-random projections followed by nonlinear (sign function) mappings. The pseudorandom projection is described by a matrix, where not all entries are independent random variables but instead a fixed "budget of randomness" is distributed across the matrix. Such matrices can be efficiently stored in sub-quadratic or even linear space, provide reduction in randomness usage (i.e. number of required random values), and very often lead to computational speed ups. We prove several theoretical results showing that projections via various structured matrices followed by nonlinear mappings accurately preserve the angular distance between input highdimensional vectors. To the best of our knowledge, these results are the first that give theoretical ground for the use of general structured matrices in the nonlinear setting. We empirically verify our theoretical findings and show the dependence of learning via structured hashed projections on the performance of neural network as well as nearest neighbor classifier.

Original languageEnglish (US)
Title of host publication33rd International Conference on Machine Learning, ICML 2016
PublisherInternational Machine Learning Society (IMLS)
Pages539-554
Number of pages16
Volume1
ISBN (Electronic)9781510829008
StatePublished - 2016
Event33rd International Conference on Machine Learning, ICML 2016 - New York City, United States
Duration: Jun 19 2016Jun 24 2016

Other

Other33rd International Conference on Machine Learning, ICML 2016
CountryUnited States
CityNew York City
Period6/19/166/24/16

    Fingerprint

ASJC Scopus subject areas

  • Artificial Intelligence
  • Software
  • Computer Networks and Communications

Cite this

Choromanska, A., Choromanski, K., Bojarski, M., Jebara, T., Kumar, S., & LeCun, Y. (2016). Binary embeddings with structured hashed projections. In 33rd International Conference on Machine Learning, ICML 2016 (Vol. 1, pp. 539-554). International Machine Learning Society (IMLS).