Learning distributed representations of sentences from unlabelled data

Felix Hill, Kyunghyun Cho, Anna Korhonen

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Unsupervised methods for learning distributed representations of words are ubiquitous in today's NLP research, but far less is known about the best ways to learn distributed phrase or sentence representations from unlabelled data. This paper is a systematic comparison of models that learn such representations. We find that the optimal approach depends critically on the intended application. Deeper, more complex models are preferable for representations to be used in supervised systems, but shallow log-bilinear models work best for building representation spaces that can be decoded with simple spatial distance metrics. We also propose two new unsupervised representation-learning objectives designed to optimise the trade-off between training time, domain portability and performance.

Original languageEnglish (US)
Title of host publication2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2016 - Proceedings of the Conference
PublisherAssociation for Computational Linguistics (ACL)
Pages1367-1377
Number of pages11
ISBN (Electronic)9781941643914
StatePublished - 2016
Event15th Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2016 - San Diego, United States
Duration: Jun 12 2016Jun 17 2016

Other

Other15th Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2016
CountryUnited States
CitySan Diego
Period6/12/166/17/16

Fingerprint

learning
learning objective
Distributed Representation
performance
Natural Language Processing
time

ASJC Scopus subject areas

  • Computer Science Applications
  • Linguistics and Language
  • Language and Linguistics

Cite this

Hill, F., Cho, K., & Korhonen, A. (2016). Learning distributed representations of sentences from unlabelled data. In 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2016 - Proceedings of the Conference (pp. 1367-1377). Association for Computational Linguistics (ACL).

Learning distributed representations of sentences from unlabelled data. / Hill, Felix; Cho, Kyunghyun; Korhonen, Anna.

2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2016 - Proceedings of the Conference. Association for Computational Linguistics (ACL), 2016. p. 1367-1377.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Hill, F, Cho, K & Korhonen, A 2016, Learning distributed representations of sentences from unlabelled data. in 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2016 - Proceedings of the Conference. Association for Computational Linguistics (ACL), pp. 1367-1377, 15th Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2016, San Diego, United States, 6/12/16.
Hill F, Cho K, Korhonen A. Learning distributed representations of sentences from unlabelled data. In 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2016 - Proceedings of the Conference. Association for Computational Linguistics (ACL). 2016. p. 1367-1377
Hill, Felix ; Cho, Kyunghyun ; Korhonen, Anna. / Learning distributed representations of sentences from unlabelled data. 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2016 - Proceedings of the Conference. Association for Computational Linguistics (ACL), 2016. pp. 1367-1377
@inproceedings{eb2fd4167c47453cb3b831a7dfd40075,
title = "Learning distributed representations of sentences from unlabelled data",
abstract = "Unsupervised methods for learning distributed representations of words are ubiquitous in today's NLP research, but far less is known about the best ways to learn distributed phrase or sentence representations from unlabelled data. This paper is a systematic comparison of models that learn such representations. We find that the optimal approach depends critically on the intended application. Deeper, more complex models are preferable for representations to be used in supervised systems, but shallow log-bilinear models work best for building representation spaces that can be decoded with simple spatial distance metrics. We also propose two new unsupervised representation-learning objectives designed to optimise the trade-off between training time, domain portability and performance.",
author = "Felix Hill and Kyunghyun Cho and Anna Korhonen",
year = "2016",
language = "English (US)",
pages = "1367--1377",
booktitle = "2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2016 - Proceedings of the Conference",
publisher = "Association for Computational Linguistics (ACL)",

}

TY - GEN

T1 - Learning distributed representations of sentences from unlabelled data

AU - Hill, Felix

AU - Cho, Kyunghyun

AU - Korhonen, Anna

PY - 2016

Y1 - 2016

N2 - Unsupervised methods for learning distributed representations of words are ubiquitous in today's NLP research, but far less is known about the best ways to learn distributed phrase or sentence representations from unlabelled data. This paper is a systematic comparison of models that learn such representations. We find that the optimal approach depends critically on the intended application. Deeper, more complex models are preferable for representations to be used in supervised systems, but shallow log-bilinear models work best for building representation spaces that can be decoded with simple spatial distance metrics. We also propose two new unsupervised representation-learning objectives designed to optimise the trade-off between training time, domain portability and performance.

AB - Unsupervised methods for learning distributed representations of words are ubiquitous in today's NLP research, but far less is known about the best ways to learn distributed phrase or sentence representations from unlabelled data. This paper is a systematic comparison of models that learn such representations. We find that the optimal approach depends critically on the intended application. Deeper, more complex models are preferable for representations to be used in supervised systems, but shallow log-bilinear models work best for building representation spaces that can be decoded with simple spatial distance metrics. We also propose two new unsupervised representation-learning objectives designed to optimise the trade-off between training time, domain portability and performance.

UR - http://www.scopus.com/inward/record.url?scp=84994157681&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84994157681&partnerID=8YFLogxK

M3 - Conference contribution

SP - 1367

EP - 1377

BT - 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2016 - Proceedings of the Conference

PB - Association for Computational Linguistics (ACL)

ER -