Attention-based models for speech recognition

Jan Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, Yoshua Bengio

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Recurrent sequence generators conditioned on input data through an attention mechanism have recently shown very good performance on a range of tasks including machine translation, handwriting synthesis [1, 2] and image caption generation [3]. We extend the attention-mechanism with features needed for speech recognition. We show that while an adaptation of the model used for machine translation in [2] reaches a competitive 18.7% phoneme error rate (PER) on the TIMIT phoneme recognition task, it can only be applied to utterances which are roughly as long as the ones it was trained on. We offer a qualitative explanation of this failure and propose a novel and generic method of adding location-awareness to the attention mechanism to alleviate this issue. The new method yields a model that is robust to long inputs and achieves 18% PER in single utterances and 20% in 10-times longer (repeated) utterances. Finally, we propose a change to the attention mechanism that prevents it from concentrating too much on single frames, which further reduces PER to 17.6% level.

Original languageEnglish (US)
Title of host publicationAdvances in Neural Information Processing Systems
PublisherNeural information processing systems foundation
Pages577-585
Number of pages9
Volume2015-January
StatePublished - 2015
Event29th Annual Conference on Neural Information Processing Systems, NIPS 2015 - Montreal, Canada
Duration: Dec 7 2015Dec 12 2015

Other

Other29th Annual Conference on Neural Information Processing Systems, NIPS 2015
CountryCanada
CityMontreal
Period12/7/1512/12/15

Fingerprint

Speech recognition

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Information Systems
  • Signal Processing

Cite this

Chorowski, J., Bahdanau, D., Serdyuk, D., Cho, K., & Bengio, Y. (2015). Attention-based models for speech recognition. In Advances in Neural Information Processing Systems (Vol. 2015-January, pp. 577-585). Neural information processing systems foundation.

Attention-based models for speech recognition. / Chorowski, Jan; Bahdanau, Dzmitry; Serdyuk, Dmitriy; Cho, Kyunghyun; Bengio, Yoshua.

Advances in Neural Information Processing Systems. Vol. 2015-January Neural information processing systems foundation, 2015. p. 577-585.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Chorowski, J, Bahdanau, D, Serdyuk, D, Cho, K & Bengio, Y 2015, Attention-based models for speech recognition. in Advances in Neural Information Processing Systems. vol. 2015-January, Neural information processing systems foundation, pp. 577-585, 29th Annual Conference on Neural Information Processing Systems, NIPS 2015, Montreal, Canada, 12/7/15.
Chorowski J, Bahdanau D, Serdyuk D, Cho K, Bengio Y. Attention-based models for speech recognition. In Advances in Neural Information Processing Systems. Vol. 2015-January. Neural information processing systems foundation. 2015. p. 577-585
Chorowski, Jan ; Bahdanau, Dzmitry ; Serdyuk, Dmitriy ; Cho, Kyunghyun ; Bengio, Yoshua. / Attention-based models for speech recognition. Advances in Neural Information Processing Systems. Vol. 2015-January Neural information processing systems foundation, 2015. pp. 577-585
@inproceedings{6ad93d9e61d44924bc97f156ddcb0923,
title = "Attention-based models for speech recognition",
abstract = "Recurrent sequence generators conditioned on input data through an attention mechanism have recently shown very good performance on a range of tasks including machine translation, handwriting synthesis [1, 2] and image caption generation [3]. We extend the attention-mechanism with features needed for speech recognition. We show that while an adaptation of the model used for machine translation in [2] reaches a competitive 18.7{\%} phoneme error rate (PER) on the TIMIT phoneme recognition task, it can only be applied to utterances which are roughly as long as the ones it was trained on. We offer a qualitative explanation of this failure and propose a novel and generic method of adding location-awareness to the attention mechanism to alleviate this issue. The new method yields a model that is robust to long inputs and achieves 18{\%} PER in single utterances and 20{\%} in 10-times longer (repeated) utterances. Finally, we propose a change to the attention mechanism that prevents it from concentrating too much on single frames, which further reduces PER to 17.6{\%} level.",
author = "Jan Chorowski and Dzmitry Bahdanau and Dmitriy Serdyuk and Kyunghyun Cho and Yoshua Bengio",
year = "2015",
language = "English (US)",
volume = "2015-January",
pages = "577--585",
booktitle = "Advances in Neural Information Processing Systems",
publisher = "Neural information processing systems foundation",

}

TY - GEN

T1 - Attention-based models for speech recognition

AU - Chorowski, Jan

AU - Bahdanau, Dzmitry

AU - Serdyuk, Dmitriy

AU - Cho, Kyunghyun

AU - Bengio, Yoshua

PY - 2015

Y1 - 2015

N2 - Recurrent sequence generators conditioned on input data through an attention mechanism have recently shown very good performance on a range of tasks including machine translation, handwriting synthesis [1, 2] and image caption generation [3]. We extend the attention-mechanism with features needed for speech recognition. We show that while an adaptation of the model used for machine translation in [2] reaches a competitive 18.7% phoneme error rate (PER) on the TIMIT phoneme recognition task, it can only be applied to utterances which are roughly as long as the ones it was trained on. We offer a qualitative explanation of this failure and propose a novel and generic method of adding location-awareness to the attention mechanism to alleviate this issue. The new method yields a model that is robust to long inputs and achieves 18% PER in single utterances and 20% in 10-times longer (repeated) utterances. Finally, we propose a change to the attention mechanism that prevents it from concentrating too much on single frames, which further reduces PER to 17.6% level.

AB - Recurrent sequence generators conditioned on input data through an attention mechanism have recently shown very good performance on a range of tasks including machine translation, handwriting synthesis [1, 2] and image caption generation [3]. We extend the attention-mechanism with features needed for speech recognition. We show that while an adaptation of the model used for machine translation in [2] reaches a competitive 18.7% phoneme error rate (PER) on the TIMIT phoneme recognition task, it can only be applied to utterances which are roughly as long as the ones it was trained on. We offer a qualitative explanation of this failure and propose a novel and generic method of adding location-awareness to the attention mechanism to alleviate this issue. The new method yields a model that is robust to long inputs and achieves 18% PER in single utterances and 20% in 10-times longer (repeated) utterances. Finally, we propose a change to the attention mechanism that prevents it from concentrating too much on single frames, which further reduces PER to 17.6% level.

UR - http://www.scopus.com/inward/record.url?scp=84965139600&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84965139600&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:84965139600

VL - 2015-January

SP - 577

EP - 585

BT - Advances in Neural Information Processing Systems

PB - Neural information processing systems foundation

ER -