Tree-structured composition in neural networks without tree-structured architectures

Samuel Bowman, Christopher D. Manning, Christopher Potts

    Research output: Contribution to journalArticle

    Abstract

    Tree-structured neural networks encode a particular tree geometry for a sentence in the network design. However, these models have at best only slightly outperformed simpler sequence-based models. We hypothesize that neural sequence models like LSTMs are in fact able to discover and implicitly use recursive compositional structure, at least for tasks with clear cues to that structure in the data. We demonstrate this possibility using an artificial data task for which recursive compositional structure is crucial, and find an LSTM-based sequence model can indeed learn to exploit the underlying tree structure. However, its performance consistently lags behind that of tree models, even on large training sets, suggesting that tree-structured models are more effective at exploiting recursive structure.

    Original languageEnglish (US)
    JournalCEUR Workshop Proceedings
    Volume1583
    StatePublished - 2015

    Fingerprint

    Neural networks
    Chemical analysis
    Geometry

    ASJC Scopus subject areas

    • Computer Science(all)

    Cite this

    Tree-structured composition in neural networks without tree-structured architectures. / Bowman, Samuel; Manning, Christopher D.; Potts, Christopher.

    In: CEUR Workshop Proceedings, Vol. 1583, 2015.

    Research output: Contribution to journalArticle

    Bowman, Samuel ; Manning, Christopher D. ; Potts, Christopher. / Tree-structured composition in neural networks without tree-structured architectures. In: CEUR Workshop Proceedings. 2015 ; Vol. 1583.
    @article{a7a61da8836c478eaacd116eef800351,
    title = "Tree-structured composition in neural networks without tree-structured architectures",
    abstract = "Tree-structured neural networks encode a particular tree geometry for a sentence in the network design. However, these models have at best only slightly outperformed simpler sequence-based models. We hypothesize that neural sequence models like LSTMs are in fact able to discover and implicitly use recursive compositional structure, at least for tasks with clear cues to that structure in the data. We demonstrate this possibility using an artificial data task for which recursive compositional structure is crucial, and find an LSTM-based sequence model can indeed learn to exploit the underlying tree structure. However, its performance consistently lags behind that of tree models, even on large training sets, suggesting that tree-structured models are more effective at exploiting recursive structure.",
    author = "Samuel Bowman and Manning, {Christopher D.} and Christopher Potts",
    year = "2015",
    language = "English (US)",
    volume = "1583",
    journal = "CEUR Workshop Proceedings",
    issn = "1613-0073",

    }

    TY - JOUR

    T1 - Tree-structured composition in neural networks without tree-structured architectures

    AU - Bowman, Samuel

    AU - Manning, Christopher D.

    AU - Potts, Christopher

    PY - 2015

    Y1 - 2015

    N2 - Tree-structured neural networks encode a particular tree geometry for a sentence in the network design. However, these models have at best only slightly outperformed simpler sequence-based models. We hypothesize that neural sequence models like LSTMs are in fact able to discover and implicitly use recursive compositional structure, at least for tasks with clear cues to that structure in the data. We demonstrate this possibility using an artificial data task for which recursive compositional structure is crucial, and find an LSTM-based sequence model can indeed learn to exploit the underlying tree structure. However, its performance consistently lags behind that of tree models, even on large training sets, suggesting that tree-structured models are more effective at exploiting recursive structure.

    AB - Tree-structured neural networks encode a particular tree geometry for a sentence in the network design. However, these models have at best only slightly outperformed simpler sequence-based models. We hypothesize that neural sequence models like LSTMs are in fact able to discover and implicitly use recursive compositional structure, at least for tasks with clear cues to that structure in the data. We demonstrate this possibility using an artificial data task for which recursive compositional structure is crucial, and find an LSTM-based sequence model can indeed learn to exploit the underlying tree structure. However, its performance consistently lags behind that of tree models, even on large training sets, suggesting that tree-structured models are more effective at exploiting recursive structure.

    UR - http://www.scopus.com/inward/record.url?scp=84977505088&partnerID=8YFLogxK

    UR - http://www.scopus.com/inward/citedby.url?scp=84977505088&partnerID=8YFLogxK

    M3 - Article

    VL - 1583

    JO - CEUR Workshop Proceedings

    JF - CEUR Workshop Proceedings

    SN - 1613-0073

    ER -