Bug synthesis

Challenging bug-finding tools with deep faults

Subhajit Roy, Awanish Pandey, Brendan Dolan-Gavitt, Yu Hu

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Abstract

    In spite of decades of research in bug detection tools, there is a surprising dearth of ground-truth corpora that can be used to evaluate the efficacy of such tools. Recently, systems such as LAVA and EvilCoder have been proposed to automatically inject bugs into software to quickly generate large bug corpora, but the bugs created so far differ from naturally occurring bugs in a number of ways. In this work, we propose a new automated bug injection system, Apocalypse, that uses formal techniques-symbolic execution, constraint-based program synthesis and model counting-to automatically inject fair (can potentially be discovered by current bug-detection tools), deep (requiring a long sequence of dependencies to be satisfied to fire), uncorrelated (each bug behaving independent of others), reproducible (a trigger input being available) and rare (can be triggered by only a few program inputs) bugs in large software code bases. In our evaluation, we inject bugs into thirty Coreutils programs as well as the TCAS test suite.We find that bugs synthesized by Apocalypse are highly realistic under a variety of metrics, that they do not favor a particular bug-finding strategy (unlike bugs produced by LAVA), and that they are more difficult to find than manually injected bugs, requiring up around 240× more tests to discover with a state-of-the-art symbolic execution tool.

    Original languageEnglish (US)
    Title of host publicationESEC/FSE 2018 - Proceedings of the 2018 26th ACM Joint Meeting on European So ftware Engineering Conference and Symposium on the Foundations of So ftware Engineering
    EditorsAlessandro Garci, Corina S. Pasareanu, Gary T. Leavens
    PublisherAssociation for Computing Machinery, Inc
    Pages224-234
    Number of pages11
    ISBN (Electronic)9781450355735
    DOIs
    StatePublished - Oct 26 2018
    Event26th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2018 - Lake Buena Vista, United States
    Duration: Nov 4 2018Nov 9 2018

    Other

    Other26th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2018
    CountryUnited States
    CityLake Buena Vista
    Period11/4/1811/9/18

    Fingerprint

    Fires

    Keywords

    • Bug Injection
    • Constraintbased Synthesis
    • Program Synthesis
    • Symbolic Execution

    ASJC Scopus subject areas

    • Software
    • Artificial Intelligence

    Cite this

    Roy, S., Pandey, A., Dolan-Gavitt, B., & Hu, Y. (2018). Bug synthesis: Challenging bug-finding tools with deep faults. In A. Garci, C. S. Pasareanu, & G. T. Leavens (Eds.), ESEC/FSE 2018 - Proceedings of the 2018 26th ACM Joint Meeting on European So ftware Engineering Conference and Symposium on the Foundations of So ftware Engineering (pp. 224-234). Association for Computing Machinery, Inc. https://doi.org/10.1145/3236024.3236084

    Bug synthesis : Challenging bug-finding tools with deep faults. / Roy, Subhajit; Pandey, Awanish; Dolan-Gavitt, Brendan; Hu, Yu.

    ESEC/FSE 2018 - Proceedings of the 2018 26th ACM Joint Meeting on European So ftware Engineering Conference and Symposium on the Foundations of So ftware Engineering. ed. / Alessandro Garci; Corina S. Pasareanu; Gary T. Leavens. Association for Computing Machinery, Inc, 2018. p. 224-234.

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Roy, S, Pandey, A, Dolan-Gavitt, B & Hu, Y 2018, Bug synthesis: Challenging bug-finding tools with deep faults. in A Garci, CS Pasareanu & GT Leavens (eds), ESEC/FSE 2018 - Proceedings of the 2018 26th ACM Joint Meeting on European So ftware Engineering Conference and Symposium on the Foundations of So ftware Engineering. Association for Computing Machinery, Inc, pp. 224-234, 26th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2018, Lake Buena Vista, United States, 11/4/18. https://doi.org/10.1145/3236024.3236084
    Roy S, Pandey A, Dolan-Gavitt B, Hu Y. Bug synthesis: Challenging bug-finding tools with deep faults. In Garci A, Pasareanu CS, Leavens GT, editors, ESEC/FSE 2018 - Proceedings of the 2018 26th ACM Joint Meeting on European So ftware Engineering Conference and Symposium on the Foundations of So ftware Engineering. Association for Computing Machinery, Inc. 2018. p. 224-234 https://doi.org/10.1145/3236024.3236084
    Roy, Subhajit ; Pandey, Awanish ; Dolan-Gavitt, Brendan ; Hu, Yu. / Bug synthesis : Challenging bug-finding tools with deep faults. ESEC/FSE 2018 - Proceedings of the 2018 26th ACM Joint Meeting on European So ftware Engineering Conference and Symposium on the Foundations of So ftware Engineering. editor / Alessandro Garci ; Corina S. Pasareanu ; Gary T. Leavens. Association for Computing Machinery, Inc, 2018. pp. 224-234
    @inproceedings{2cdc29d13a9143ec8ebab11fd874d2a8,
    title = "Bug synthesis: Challenging bug-finding tools with deep faults",
    abstract = "In spite of decades of research in bug detection tools, there is a surprising dearth of ground-truth corpora that can be used to evaluate the efficacy of such tools. Recently, systems such as LAVA and EvilCoder have been proposed to automatically inject bugs into software to quickly generate large bug corpora, but the bugs created so far differ from naturally occurring bugs in a number of ways. In this work, we propose a new automated bug injection system, Apocalypse, that uses formal techniques-symbolic execution, constraint-based program synthesis and model counting-to automatically inject fair (can potentially be discovered by current bug-detection tools), deep (requiring a long sequence of dependencies to be satisfied to fire), uncorrelated (each bug behaving independent of others), reproducible (a trigger input being available) and rare (can be triggered by only a few program inputs) bugs in large software code bases. In our evaluation, we inject bugs into thirty Coreutils programs as well as the TCAS test suite.We find that bugs synthesized by Apocalypse are highly realistic under a variety of metrics, that they do not favor a particular bug-finding strategy (unlike bugs produced by LAVA), and that they are more difficult to find than manually injected bugs, requiring up around 240× more tests to discover with a state-of-the-art symbolic execution tool.",
    keywords = "Bug Injection, Constraintbased Synthesis, Program Synthesis, Symbolic Execution",
    author = "Subhajit Roy and Awanish Pandey and Brendan Dolan-Gavitt and Yu Hu",
    year = "2018",
    month = "10",
    day = "26",
    doi = "10.1145/3236024.3236084",
    language = "English (US)",
    pages = "224--234",
    editor = "Alessandro Garci and Pasareanu, {Corina S.} and Leavens, {Gary T.}",
    booktitle = "ESEC/FSE 2018 - Proceedings of the 2018 26th ACM Joint Meeting on European So ftware Engineering Conference and Symposium on the Foundations of So ftware Engineering",
    publisher = "Association for Computing Machinery, Inc",

    }

    TY - GEN

    T1 - Bug synthesis

    T2 - Challenging bug-finding tools with deep faults

    AU - Roy, Subhajit

    AU - Pandey, Awanish

    AU - Dolan-Gavitt, Brendan

    AU - Hu, Yu

    PY - 2018/10/26

    Y1 - 2018/10/26

    N2 - In spite of decades of research in bug detection tools, there is a surprising dearth of ground-truth corpora that can be used to evaluate the efficacy of such tools. Recently, systems such as LAVA and EvilCoder have been proposed to automatically inject bugs into software to quickly generate large bug corpora, but the bugs created so far differ from naturally occurring bugs in a number of ways. In this work, we propose a new automated bug injection system, Apocalypse, that uses formal techniques-symbolic execution, constraint-based program synthesis and model counting-to automatically inject fair (can potentially be discovered by current bug-detection tools), deep (requiring a long sequence of dependencies to be satisfied to fire), uncorrelated (each bug behaving independent of others), reproducible (a trigger input being available) and rare (can be triggered by only a few program inputs) bugs in large software code bases. In our evaluation, we inject bugs into thirty Coreutils programs as well as the TCAS test suite.We find that bugs synthesized by Apocalypse are highly realistic under a variety of metrics, that they do not favor a particular bug-finding strategy (unlike bugs produced by LAVA), and that they are more difficult to find than manually injected bugs, requiring up around 240× more tests to discover with a state-of-the-art symbolic execution tool.

    AB - In spite of decades of research in bug detection tools, there is a surprising dearth of ground-truth corpora that can be used to evaluate the efficacy of such tools. Recently, systems such as LAVA and EvilCoder have been proposed to automatically inject bugs into software to quickly generate large bug corpora, but the bugs created so far differ from naturally occurring bugs in a number of ways. In this work, we propose a new automated bug injection system, Apocalypse, that uses formal techniques-symbolic execution, constraint-based program synthesis and model counting-to automatically inject fair (can potentially be discovered by current bug-detection tools), deep (requiring a long sequence of dependencies to be satisfied to fire), uncorrelated (each bug behaving independent of others), reproducible (a trigger input being available) and rare (can be triggered by only a few program inputs) bugs in large software code bases. In our evaluation, we inject bugs into thirty Coreutils programs as well as the TCAS test suite.We find that bugs synthesized by Apocalypse are highly realistic under a variety of metrics, that they do not favor a particular bug-finding strategy (unlike bugs produced by LAVA), and that they are more difficult to find than manually injected bugs, requiring up around 240× more tests to discover with a state-of-the-art symbolic execution tool.

    KW - Bug Injection

    KW - Constraintbased Synthesis

    KW - Program Synthesis

    KW - Symbolic Execution

    UR - http://www.scopus.com/inward/record.url?scp=85058301253&partnerID=8YFLogxK

    UR - http://www.scopus.com/inward/citedby.url?scp=85058301253&partnerID=8YFLogxK

    U2 - 10.1145/3236024.3236084

    DO - 10.1145/3236024.3236084

    M3 - Conference contribution

    SP - 224

    EP - 234

    BT - ESEC/FSE 2018 - Proceedings of the 2018 26th ACM Joint Meeting on European So ftware Engineering Conference and Symposium on the Foundations of So ftware Engineering

    A2 - Garci, Alessandro

    A2 - Pasareanu, Corina S.

    A2 - Leavens, Gary T.

    PB - Association for Computing Machinery, Inc

    ER -