Character-Aware neural language models

Yoon Kim, Yacine Jernite, David Sontag, Alexander M. Rush

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

We describe a simple neural language model that relies only on character-level inputs. Predictions are still made at the word-level. Our model employs a convolutional neural network (CNN) and a highway network over characters, whose output is given to a long short-Term memory (LSTM) recurrent neural network language model (RNN-LM). On the English Penn Treebank the model is on par with the existing state-of-The-Art despite having 60% fewer parameters. On languages with rich morphology (Arabic, Czech, French, German, Spanish, Russian), the model outperforms word-level/morpheme-level LSTM baselines, again with fewer parameters. The results suggest that on many languages, character inputs are sufficient for language modeling. Analysis of word representations obtained from the character composition part of the model reveals that the model is able to encode, from characters only, both semantic and orthographic information.

Original languageEnglish (US)
Title of host publication30th AAAI Conference on Artificial Intelligence, AAAI 2016
PublisherAAAI press
Pages2741-2749
Number of pages9
ISBN (Electronic)9781577357605
StatePublished - 2016
Event30th AAAI Conference on Artificial Intelligence, AAAI 2016 - Phoenix, United States
Duration: Feb 12 2016Feb 17 2016

Other

Other30th AAAI Conference on Artificial Intelligence, AAAI 2016
CountryUnited States
CityPhoenix
Period2/12/162/17/16

Fingerprint

Recurrent neural networks
Semantics
Neural networks
Chemical analysis
Long short-term memory

ASJC Scopus subject areas

  • Artificial Intelligence

Cite this

Kim, Y., Jernite, Y., Sontag, D., & Rush, A. M. (2016). Character-Aware neural language models. In 30th AAAI Conference on Artificial Intelligence, AAAI 2016 (pp. 2741-2749). AAAI press.

Character-Aware neural language models. / Kim, Yoon; Jernite, Yacine; Sontag, David; Rush, Alexander M.

30th AAAI Conference on Artificial Intelligence, AAAI 2016. AAAI press, 2016. p. 2741-2749.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Kim, Y, Jernite, Y, Sontag, D & Rush, AM 2016, Character-Aware neural language models. in 30th AAAI Conference on Artificial Intelligence, AAAI 2016. AAAI press, pp. 2741-2749, 30th AAAI Conference on Artificial Intelligence, AAAI 2016, Phoenix, United States, 2/12/16.
Kim Y, Jernite Y, Sontag D, Rush AM. Character-Aware neural language models. In 30th AAAI Conference on Artificial Intelligence, AAAI 2016. AAAI press. 2016. p. 2741-2749
Kim, Yoon ; Jernite, Yacine ; Sontag, David ; Rush, Alexander M. / Character-Aware neural language models. 30th AAAI Conference on Artificial Intelligence, AAAI 2016. AAAI press, 2016. pp. 2741-2749
@inproceedings{8b788dae3ed240d29fe02b54ea3c6dc4,
title = "Character-Aware neural language models",
abstract = "We describe a simple neural language model that relies only on character-level inputs. Predictions are still made at the word-level. Our model employs a convolutional neural network (CNN) and a highway network over characters, whose output is given to a long short-Term memory (LSTM) recurrent neural network language model (RNN-LM). On the English Penn Treebank the model is on par with the existing state-of-The-Art despite having 60{\%} fewer parameters. On languages with rich morphology (Arabic, Czech, French, German, Spanish, Russian), the model outperforms word-level/morpheme-level LSTM baselines, again with fewer parameters. The results suggest that on many languages, character inputs are sufficient for language modeling. Analysis of word representations obtained from the character composition part of the model reveals that the model is able to encode, from characters only, both semantic and orthographic information.",
author = "Yoon Kim and Yacine Jernite and David Sontag and Rush, {Alexander M.}",
year = "2016",
language = "English (US)",
pages = "2741--2749",
booktitle = "30th AAAI Conference on Artificial Intelligence, AAAI 2016",
publisher = "AAAI press",

}

TY - GEN

T1 - Character-Aware neural language models

AU - Kim, Yoon

AU - Jernite, Yacine

AU - Sontag, David

AU - Rush, Alexander M.

PY - 2016

Y1 - 2016

N2 - We describe a simple neural language model that relies only on character-level inputs. Predictions are still made at the word-level. Our model employs a convolutional neural network (CNN) and a highway network over characters, whose output is given to a long short-Term memory (LSTM) recurrent neural network language model (RNN-LM). On the English Penn Treebank the model is on par with the existing state-of-The-Art despite having 60% fewer parameters. On languages with rich morphology (Arabic, Czech, French, German, Spanish, Russian), the model outperforms word-level/morpheme-level LSTM baselines, again with fewer parameters. The results suggest that on many languages, character inputs are sufficient for language modeling. Analysis of word representations obtained from the character composition part of the model reveals that the model is able to encode, from characters only, both semantic and orthographic information.

AB - We describe a simple neural language model that relies only on character-level inputs. Predictions are still made at the word-level. Our model employs a convolutional neural network (CNN) and a highway network over characters, whose output is given to a long short-Term memory (LSTM) recurrent neural network language model (RNN-LM). On the English Penn Treebank the model is on par with the existing state-of-The-Art despite having 60% fewer parameters. On languages with rich morphology (Arabic, Czech, French, German, Spanish, Russian), the model outperforms word-level/morpheme-level LSTM baselines, again with fewer parameters. The results suggest that on many languages, character inputs are sufficient for language modeling. Analysis of word representations obtained from the character composition part of the model reveals that the model is able to encode, from characters only, both semantic and orthographic information.

UR - http://www.scopus.com/inward/record.url?scp=84996482260&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84996482260&partnerID=8YFLogxK

M3 - Conference contribution

SP - 2741

EP - 2749

BT - 30th AAAI Conference on Artificial Intelligence, AAAI 2016

PB - AAAI press

ER -