Recursive Feature Elimination by Sensitivity Testing

Nicholas Sean Escanilla, Lisa Hellerstein, Ross Kleiman, Zhaobin Kuang, James Shull, David Page

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Abstract

    There is great interest in methods to improve human insight into trained non-linear models. Leading approaches include producing a ranking of the most relevant features, a non-trivial task for non-linear models. We show theoretically and empirically the benefit of a novel version of recursive feature elimination (RFE) as often used with SVMs; the key idea is a simple twist on the kinds of sensitivity testing employed in computational learning theory with membership queries (e.g., [1]). With membership queries, one can check whether changing the value of a feature in an example changes the label. In the real-world, we usually cannot get answers to such queries, so our approach instead makes these queries to a trained (imperfect) non-linear model. Because SVMs are widely used in bioinformatics, our empirical results use a real-world cancer genomics problem; because ground truth is not known for this task, we discuss the potential insights provided. We also evaluate on synthetic data where ground truth is known.

    Original languageEnglish (US)
    Title of host publicationProceedings - 17th IEEE International Conference on Machine Learning and Applications, ICMLA 2018
    EditorsM. Arif Wani, Moamar Sayed-Mouchaweh, Edwin Lughofer, Joao Gama, Mehmed Kantardzic
    PublisherInstitute of Electrical and Electronics Engineers Inc.
    Pages40-47
    Number of pages8
    ISBN (Electronic)9781538668047
    DOIs
    StatePublished - Jan 15 2019
    Event17th IEEE International Conference on Machine Learning and Applications, ICMLA 2018 - Orlando, United States
    Duration: Dec 17 2018Dec 20 2018

    Publication series

    NameProceedings - 17th IEEE International Conference on Machine Learning and Applications, ICMLA 2018

    Conference

    Conference17th IEEE International Conference on Machine Learning and Applications, ICMLA 2018
    CountryUnited States
    CityOrlando
    Period12/17/1812/20/18

    Fingerprint

    Testing
    Bioinformatics
    Labels
    Query
    Genomics
    Learning theory
    Cancer
    Ranking
    Twist
    Empirical results

    Keywords

    • Correlation immunity
    • Feature ranking
    • Feature selection
    • Machine learning

    ASJC Scopus subject areas

    • Artificial Intelligence
    • Computer Networks and Communications
    • Computer Science Applications
    • Computer Vision and Pattern Recognition
    • Safety, Risk, Reliability and Quality
    • Signal Processing
    • Decision Sciences (miscellaneous)

    Cite this

    Escanilla, N. S., Hellerstein, L., Kleiman, R., Kuang, Z., Shull, J., & Page, D. (2019). Recursive Feature Elimination by Sensitivity Testing. In M. A. Wani, M. Sayed-Mouchaweh, E. Lughofer, J. Gama, & M. Kantardzic (Eds.), Proceedings - 17th IEEE International Conference on Machine Learning and Applications, ICMLA 2018 (pp. 40-47). [8614039] (Proceedings - 17th IEEE International Conference on Machine Learning and Applications, ICMLA 2018). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICMLA.2018.00014

    Recursive Feature Elimination by Sensitivity Testing. / Escanilla, Nicholas Sean; Hellerstein, Lisa; Kleiman, Ross; Kuang, Zhaobin; Shull, James; Page, David.

    Proceedings - 17th IEEE International Conference on Machine Learning and Applications, ICMLA 2018. ed. / M. Arif Wani; Moamar Sayed-Mouchaweh; Edwin Lughofer; Joao Gama; Mehmed Kantardzic. Institute of Electrical and Electronics Engineers Inc., 2019. p. 40-47 8614039 (Proceedings - 17th IEEE International Conference on Machine Learning and Applications, ICMLA 2018).

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Escanilla, NS, Hellerstein, L, Kleiman, R, Kuang, Z, Shull, J & Page, D 2019, Recursive Feature Elimination by Sensitivity Testing. in MA Wani, M Sayed-Mouchaweh, E Lughofer, J Gama & M Kantardzic (eds), Proceedings - 17th IEEE International Conference on Machine Learning and Applications, ICMLA 2018., 8614039, Proceedings - 17th IEEE International Conference on Machine Learning and Applications, ICMLA 2018, Institute of Electrical and Electronics Engineers Inc., pp. 40-47, 17th IEEE International Conference on Machine Learning and Applications, ICMLA 2018, Orlando, United States, 12/17/18. https://doi.org/10.1109/ICMLA.2018.00014
    Escanilla NS, Hellerstein L, Kleiman R, Kuang Z, Shull J, Page D. Recursive Feature Elimination by Sensitivity Testing. In Wani MA, Sayed-Mouchaweh M, Lughofer E, Gama J, Kantardzic M, editors, Proceedings - 17th IEEE International Conference on Machine Learning and Applications, ICMLA 2018. Institute of Electrical and Electronics Engineers Inc. 2019. p. 40-47. 8614039. (Proceedings - 17th IEEE International Conference on Machine Learning and Applications, ICMLA 2018). https://doi.org/10.1109/ICMLA.2018.00014
    Escanilla, Nicholas Sean ; Hellerstein, Lisa ; Kleiman, Ross ; Kuang, Zhaobin ; Shull, James ; Page, David. / Recursive Feature Elimination by Sensitivity Testing. Proceedings - 17th IEEE International Conference on Machine Learning and Applications, ICMLA 2018. editor / M. Arif Wani ; Moamar Sayed-Mouchaweh ; Edwin Lughofer ; Joao Gama ; Mehmed Kantardzic. Institute of Electrical and Electronics Engineers Inc., 2019. pp. 40-47 (Proceedings - 17th IEEE International Conference on Machine Learning and Applications, ICMLA 2018).
    @inproceedings{ed462e0c29cb4f67bd486189d4a77e41,
    title = "Recursive Feature Elimination by Sensitivity Testing",
    abstract = "There is great interest in methods to improve human insight into trained non-linear models. Leading approaches include producing a ranking of the most relevant features, a non-trivial task for non-linear models. We show theoretically and empirically the benefit of a novel version of recursive feature elimination (RFE) as often used with SVMs; the key idea is a simple twist on the kinds of sensitivity testing employed in computational learning theory with membership queries (e.g., [1]). With membership queries, one can check whether changing the value of a feature in an example changes the label. In the real-world, we usually cannot get answers to such queries, so our approach instead makes these queries to a trained (imperfect) non-linear model. Because SVMs are widely used in bioinformatics, our empirical results use a real-world cancer genomics problem; because ground truth is not known for this task, we discuss the potential insights provided. We also evaluate on synthetic data where ground truth is known.",
    keywords = "Correlation immunity, Feature ranking, Feature selection, Machine learning",
    author = "Escanilla, {Nicholas Sean} and Lisa Hellerstein and Ross Kleiman and Zhaobin Kuang and James Shull and David Page",
    year = "2019",
    month = "1",
    day = "15",
    doi = "10.1109/ICMLA.2018.00014",
    language = "English (US)",
    series = "Proceedings - 17th IEEE International Conference on Machine Learning and Applications, ICMLA 2018",
    publisher = "Institute of Electrical and Electronics Engineers Inc.",
    pages = "40--47",
    editor = "Wani, {M. Arif} and Moamar Sayed-Mouchaweh and Edwin Lughofer and Joao Gama and Mehmed Kantardzic",
    booktitle = "Proceedings - 17th IEEE International Conference on Machine Learning and Applications, ICMLA 2018",

    }

    TY - GEN

    T1 - Recursive Feature Elimination by Sensitivity Testing

    AU - Escanilla, Nicholas Sean

    AU - Hellerstein, Lisa

    AU - Kleiman, Ross

    AU - Kuang, Zhaobin

    AU - Shull, James

    AU - Page, David

    PY - 2019/1/15

    Y1 - 2019/1/15

    N2 - There is great interest in methods to improve human insight into trained non-linear models. Leading approaches include producing a ranking of the most relevant features, a non-trivial task for non-linear models. We show theoretically and empirically the benefit of a novel version of recursive feature elimination (RFE) as often used with SVMs; the key idea is a simple twist on the kinds of sensitivity testing employed in computational learning theory with membership queries (e.g., [1]). With membership queries, one can check whether changing the value of a feature in an example changes the label. In the real-world, we usually cannot get answers to such queries, so our approach instead makes these queries to a trained (imperfect) non-linear model. Because SVMs are widely used in bioinformatics, our empirical results use a real-world cancer genomics problem; because ground truth is not known for this task, we discuss the potential insights provided. We also evaluate on synthetic data where ground truth is known.

    AB - There is great interest in methods to improve human insight into trained non-linear models. Leading approaches include producing a ranking of the most relevant features, a non-trivial task for non-linear models. We show theoretically and empirically the benefit of a novel version of recursive feature elimination (RFE) as often used with SVMs; the key idea is a simple twist on the kinds of sensitivity testing employed in computational learning theory with membership queries (e.g., [1]). With membership queries, one can check whether changing the value of a feature in an example changes the label. In the real-world, we usually cannot get answers to such queries, so our approach instead makes these queries to a trained (imperfect) non-linear model. Because SVMs are widely used in bioinformatics, our empirical results use a real-world cancer genomics problem; because ground truth is not known for this task, we discuss the potential insights provided. We also evaluate on synthetic data where ground truth is known.

    KW - Correlation immunity

    KW - Feature ranking

    KW - Feature selection

    KW - Machine learning

    UR - http://www.scopus.com/inward/record.url?scp=85062219541&partnerID=8YFLogxK

    UR - http://www.scopus.com/inward/citedby.url?scp=85062219541&partnerID=8YFLogxK

    U2 - 10.1109/ICMLA.2018.00014

    DO - 10.1109/ICMLA.2018.00014

    M3 - Conference contribution

    AN - SCOPUS:85062219541

    T3 - Proceedings - 17th IEEE International Conference on Machine Learning and Applications, ICMLA 2018

    SP - 40

    EP - 47

    BT - Proceedings - 17th IEEE International Conference on Machine Learning and Applications, ICMLA 2018

    A2 - Wani, M. Arif

    A2 - Sayed-Mouchaweh, Moamar

    A2 - Lughofer, Edwin

    A2 - Gama, Joao

    A2 - Kantardzic, Mehmed

    PB - Institute of Electrical and Electronics Engineers Inc.

    ER -