Further empirical studies of test effectiveness

Phyllis Frankl, Oleg Iakounenko

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Abstract

    This paper reports on an empirical evaluation of the fault-detecting ability of two white-box software testing techniques: decision coverage (branch testing) and the all-uses data flow testing criterion. Each subject program was tested using a very large number of randomly generated test sets. For each test set, the extent to which it satisfied the given testing criterion was measured and it was determined whether or not the test set detected a program fault. These data were used to explore the relationship between the coverage achieved by test sets and the likelihood that they will detect a fault. Previous experiments of this nature have used relatively small subject programs and/or have used programs with seeded faults. In contrast, the subjects used here were eight versions of an antenna configuration program written for the European Space Agency, each consisting of over 10,000 lines of C code. For each of the subject programs studied, the likelihood of detecting a fault increased sharply as very high coverage levels were reached. Thus, this data supports the belief that these testing techniques can be more effective than random testing. However, the magnitudes of the increases were rather inconsistent and it was difficult to achieve high coverage levels.

    Original languageEnglish (US)
    Title of host publicationProceedings of the ACM SIGSOFT Symposium on the Foundations of Software Engineering
    PublisherACM
    Pages153-162
    Number of pages10
    StatePublished - 1998
    EventProceedings of the 1998 ACM SIGSOFT 6th International Symposium on the Foundations of Software Engineering, FSE-6, SIGSOFT-98 - Lake Buena Vista, FL, USA
    Duration: Nov 3 1998Nov 5 1998

    Other

    OtherProceedings of the 1998 ACM SIGSOFT 6th International Symposium on the Foundations of Software Engineering, FSE-6, SIGSOFT-98
    CityLake Buena Vista, FL, USA
    Period11/3/9811/5/98

    Fingerprint

    Testing
    Software testing
    Antennas
    Experiments

    ASJC Scopus subject areas

    • Computer Science(all)

    Cite this

    Frankl, P., & Iakounenko, O. (1998). Further empirical studies of test effectiveness. In Proceedings of the ACM SIGSOFT Symposium on the Foundations of Software Engineering (pp. 153-162). ACM.

    Further empirical studies of test effectiveness. / Frankl, Phyllis; Iakounenko, Oleg.

    Proceedings of the ACM SIGSOFT Symposium on the Foundations of Software Engineering. ACM, 1998. p. 153-162.

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Frankl, P & Iakounenko, O 1998, Further empirical studies of test effectiveness. in Proceedings of the ACM SIGSOFT Symposium on the Foundations of Software Engineering. ACM, pp. 153-162, Proceedings of the 1998 ACM SIGSOFT 6th International Symposium on the Foundations of Software Engineering, FSE-6, SIGSOFT-98, Lake Buena Vista, FL, USA, 11/3/98.
    Frankl P, Iakounenko O. Further empirical studies of test effectiveness. In Proceedings of the ACM SIGSOFT Symposium on the Foundations of Software Engineering. ACM. 1998. p. 153-162
    Frankl, Phyllis ; Iakounenko, Oleg. / Further empirical studies of test effectiveness. Proceedings of the ACM SIGSOFT Symposium on the Foundations of Software Engineering. ACM, 1998. pp. 153-162
    @inproceedings{8bbe15bab4154a5b82971e7b4b21ae75,
    title = "Further empirical studies of test effectiveness",
    abstract = "This paper reports on an empirical evaluation of the fault-detecting ability of two white-box software testing techniques: decision coverage (branch testing) and the all-uses data flow testing criterion. Each subject program was tested using a very large number of randomly generated test sets. For each test set, the extent to which it satisfied the given testing criterion was measured and it was determined whether or not the test set detected a program fault. These data were used to explore the relationship between the coverage achieved by test sets and the likelihood that they will detect a fault. Previous experiments of this nature have used relatively small subject programs and/or have used programs with seeded faults. In contrast, the subjects used here were eight versions of an antenna configuration program written for the European Space Agency, each consisting of over 10,000 lines of C code. For each of the subject programs studied, the likelihood of detecting a fault increased sharply as very high coverage levels were reached. Thus, this data supports the belief that these testing techniques can be more effective than random testing. However, the magnitudes of the increases were rather inconsistent and it was difficult to achieve high coverage levels.",
    author = "Phyllis Frankl and Oleg Iakounenko",
    year = "1998",
    language = "English (US)",
    pages = "153--162",
    booktitle = "Proceedings of the ACM SIGSOFT Symposium on the Foundations of Software Engineering",
    publisher = "ACM",

    }

    TY - GEN

    T1 - Further empirical studies of test effectiveness

    AU - Frankl, Phyllis

    AU - Iakounenko, Oleg

    PY - 1998

    Y1 - 1998

    N2 - This paper reports on an empirical evaluation of the fault-detecting ability of two white-box software testing techniques: decision coverage (branch testing) and the all-uses data flow testing criterion. Each subject program was tested using a very large number of randomly generated test sets. For each test set, the extent to which it satisfied the given testing criterion was measured and it was determined whether or not the test set detected a program fault. These data were used to explore the relationship between the coverage achieved by test sets and the likelihood that they will detect a fault. Previous experiments of this nature have used relatively small subject programs and/or have used programs with seeded faults. In contrast, the subjects used here were eight versions of an antenna configuration program written for the European Space Agency, each consisting of over 10,000 lines of C code. For each of the subject programs studied, the likelihood of detecting a fault increased sharply as very high coverage levels were reached. Thus, this data supports the belief that these testing techniques can be more effective than random testing. However, the magnitudes of the increases were rather inconsistent and it was difficult to achieve high coverage levels.

    AB - This paper reports on an empirical evaluation of the fault-detecting ability of two white-box software testing techniques: decision coverage (branch testing) and the all-uses data flow testing criterion. Each subject program was tested using a very large number of randomly generated test sets. For each test set, the extent to which it satisfied the given testing criterion was measured and it was determined whether or not the test set detected a program fault. These data were used to explore the relationship between the coverage achieved by test sets and the likelihood that they will detect a fault. Previous experiments of this nature have used relatively small subject programs and/or have used programs with seeded faults. In contrast, the subjects used here were eight versions of an antenna configuration program written for the European Space Agency, each consisting of over 10,000 lines of C code. For each of the subject programs studied, the likelihood of detecting a fault increased sharply as very high coverage levels were reached. Thus, this data supports the belief that these testing techniques can be more effective than random testing. However, the magnitudes of the increases were rather inconsistent and it was difficult to achieve high coverage levels.

    UR - http://www.scopus.com/inward/record.url?scp=0032218212&partnerID=8YFLogxK

    UR - http://www.scopus.com/inward/citedby.url?scp=0032218212&partnerID=8YFLogxK

    M3 - Conference contribution

    SP - 153

    EP - 162

    BT - Proceedings of the ACM SIGSOFT Symposium on the Foundations of Software Engineering

    PB - ACM

    ER -