What do you learn from context? Probing for sentence structure in contextualized word representations

Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel Bowman, Dipanjan Das, Ellie Pavlick

    Research output: Contribution to conferencePaper

    Abstract

    Contextualized representation models such as ELMo (Peters et al., 2018a) and BERT (Devlin et al., 2018) have recently achieved state-of-the-art results on a diverse array of downstream NLP tasks. Building on recent token-level probing work, we introduce a novel edge probing task design and construct a broad suite of sub-sentence tasks derived from the traditional structured NLP pipeline. We probe word-level contextual representations from four recent models and investigate how they encode sentence structure across a range of syntactic, semantic, local, and long-range phenomena. We find that existing models trained on language modeling and translation produce strong representations for syntactic phenomena, but only offer comparably small improvements on semantic tasks over a non-contextual baseline.

    Original languageEnglish (US)
    StatePublished - Jan 1 2019
    Event7th International Conference on Learning Representations, ICLR 2019 - New Orleans, United States
    Duration: May 6 2019May 9 2019

    Conference

    Conference7th International Conference on Learning Representations, ICLR 2019
    CountryUnited States
    CityNew Orleans
    Period5/6/195/9/19

    Fingerprint

    Syntactics
    Semantics
    semantics
    Pipelines
    language
    Natural Language Processing
    Syntax
    Contextual
    Task Design
    Language Modeling

    ASJC Scopus subject areas

    • Education
    • Computer Science Applications
    • Linguistics and Language
    • Language and Linguistics

    Cite this

    Tenney, I., Xia, P., Chen, B., Wang, A., Poliak, A., Thomas McCoy, R., ... Pavlick, E. (2019). What do you learn from context? Probing for sentence structure in contextualized word representations. Paper presented at 7th International Conference on Learning Representations, ICLR 2019, New Orleans, United States.

    What do you learn from context? Probing for sentence structure in contextualized word representations. / Tenney, Ian; Xia, Patrick; Chen, Berlin; Wang, Alex; Poliak, Adam; Thomas McCoy, R.; Kim, Najoung; Van Durme, Benjamin; Bowman, Samuel; Das, Dipanjan; Pavlick, Ellie.

    2019. Paper presented at 7th International Conference on Learning Representations, ICLR 2019, New Orleans, United States.

    Research output: Contribution to conferencePaper

    Tenney, I, Xia, P, Chen, B, Wang, A, Poliak, A, Thomas McCoy, R, Kim, N, Van Durme, B, Bowman, S, Das, D & Pavlick, E 2019, 'What do you learn from context? Probing for sentence structure in contextualized word representations' Paper presented at 7th International Conference on Learning Representations, ICLR 2019, New Orleans, United States, 5/6/19 - 5/9/19, .
    Tenney I, Xia P, Chen B, Wang A, Poliak A, Thomas McCoy R et al. What do you learn from context? Probing for sentence structure in contextualized word representations. 2019. Paper presented at 7th International Conference on Learning Representations, ICLR 2019, New Orleans, United States.
    Tenney, Ian ; Xia, Patrick ; Chen, Berlin ; Wang, Alex ; Poliak, Adam ; Thomas McCoy, R. ; Kim, Najoung ; Van Durme, Benjamin ; Bowman, Samuel ; Das, Dipanjan ; Pavlick, Ellie. / What do you learn from context? Probing for sentence structure in contextualized word representations. Paper presented at 7th International Conference on Learning Representations, ICLR 2019, New Orleans, United States.
    @conference{7d48ec9ae06d402c894e3bea30841e33,
    title = "What do you learn from context? Probing for sentence structure in contextualized word representations",
    abstract = "Contextualized representation models such as ELMo (Peters et al., 2018a) and BERT (Devlin et al., 2018) have recently achieved state-of-the-art results on a diverse array of downstream NLP tasks. Building on recent token-level probing work, we introduce a novel edge probing task design and construct a broad suite of sub-sentence tasks derived from the traditional structured NLP pipeline. We probe word-level contextual representations from four recent models and investigate how they encode sentence structure across a range of syntactic, semantic, local, and long-range phenomena. We find that existing models trained on language modeling and translation produce strong representations for syntactic phenomena, but only offer comparably small improvements on semantic tasks over a non-contextual baseline.",
    author = "Ian Tenney and Patrick Xia and Berlin Chen and Alex Wang and Adam Poliak and {Thomas McCoy}, R. and Najoung Kim and {Van Durme}, Benjamin and Samuel Bowman and Dipanjan Das and Ellie Pavlick",
    year = "2019",
    month = "1",
    day = "1",
    language = "English (US)",
    note = "7th International Conference on Learning Representations, ICLR 2019 ; Conference date: 06-05-2019 Through 09-05-2019",

    }

    TY - CONF

    T1 - What do you learn from context? Probing for sentence structure in contextualized word representations

    AU - Tenney, Ian

    AU - Xia, Patrick

    AU - Chen, Berlin

    AU - Wang, Alex

    AU - Poliak, Adam

    AU - Thomas McCoy, R.

    AU - Kim, Najoung

    AU - Van Durme, Benjamin

    AU - Bowman, Samuel

    AU - Das, Dipanjan

    AU - Pavlick, Ellie

    PY - 2019/1/1

    Y1 - 2019/1/1

    N2 - Contextualized representation models such as ELMo (Peters et al., 2018a) and BERT (Devlin et al., 2018) have recently achieved state-of-the-art results on a diverse array of downstream NLP tasks. Building on recent token-level probing work, we introduce a novel edge probing task design and construct a broad suite of sub-sentence tasks derived from the traditional structured NLP pipeline. We probe word-level contextual representations from four recent models and investigate how they encode sentence structure across a range of syntactic, semantic, local, and long-range phenomena. We find that existing models trained on language modeling and translation produce strong representations for syntactic phenomena, but only offer comparably small improvements on semantic tasks over a non-contextual baseline.

    AB - Contextualized representation models such as ELMo (Peters et al., 2018a) and BERT (Devlin et al., 2018) have recently achieved state-of-the-art results on a diverse array of downstream NLP tasks. Building on recent token-level probing work, we introduce a novel edge probing task design and construct a broad suite of sub-sentence tasks derived from the traditional structured NLP pipeline. We probe word-level contextual representations from four recent models and investigate how they encode sentence structure across a range of syntactic, semantic, local, and long-range phenomena. We find that existing models trained on language modeling and translation produce strong representations for syntactic phenomena, but only offer comparably small improvements on semantic tasks over a non-contextual baseline.

    UR - http://www.scopus.com/inward/record.url?scp=85071171622&partnerID=8YFLogxK

    UR - http://www.scopus.com/inward/citedby.url?scp=85071171622&partnerID=8YFLogxK

    M3 - Paper

    ER -