Complexity theoretic hardness results for query learning

Howard Aizenstein, Tibor Hegedus, Lisa Hellerstein, Leonard Pitt

    Research output: Contribution to journalArticle

    Abstract

    We investigate the complexity of learning for the well-studied model in which the learning algorithm may ask membership and equivalence queries. While complexity theoretic techniques have previously been used to prove hardness results in various learning models, these techniques typically are not strong enough to use when a learning algorithm may make membership queries. We develop a general technique for proving hardness results for learning with membership and equivalence queries (and for more general query models). We apply the technique to show that, assuming NP ≠ co-NP, no polynomial-time membership and (proper) equivalence query algorithms exist for exactly learning read-thrice DNF formulas, unions of k ≥ 3 halfspaces over the Boolean domain, or some other related classes. Our hardness results are representation dependent, and do not preclude the existence of representation independent algorithms. The general technique introduces the representation problem for a class F of representations (e.g., formulas), which is naturally associated with the learning problem for F. This problem is related to the structural question of how to characterize functions representable by formulas in F, and is a generalization of standard complexity problems such as SATISFIABILITY. While in general the representation problem is in ΣP2, we present a theorem demonstrating that for "reasonable" classes F, the existence of a polynomial-time membership and equivalence query algorithm for exactly learning F implies that the representation problem for F is in fact in co-NP. The theorem is applied to prove hardness results such as the ones mentioned above, by showing that the representation problem for specific classes of formulas is NP-hard.

    Original languageEnglish (US)
    Pages (from-to)19-53
    Number of pages35
    JournalComputational Complexity
    Volume7
    Issue number1
    StatePublished - 1998

    Fingerprint

    Hardness
    Query
    Learning algorithms
    Equivalence
    Polynomials
    Learning Algorithm
    Polynomial time
    Representation Formula
    Learning
    Theorem
    Half-space
    Union
    NP-complete problem
    Model
    Imply
    Class
    Dependent

    Keywords

    • Complexity theory
    • Computational learning theory
    • Equivalence queries
    • Membership queries
    • Query learning
    • Read-thrice DNF
    • Threshold functions

    ASJC Scopus subject areas

    • Computational Theory and Mathematics
    • Mathematics(all)
    • Computational Mathematics
    • Theoretical Computer Science

    Cite this

    Aizenstein, H., Hegedus, T., Hellerstein, L., & Pitt, L. (1998). Complexity theoretic hardness results for query learning. Computational Complexity, 7(1), 19-53.

    Complexity theoretic hardness results for query learning. / Aizenstein, Howard; Hegedus, Tibor; Hellerstein, Lisa; Pitt, Leonard.

    In: Computational Complexity, Vol. 7, No. 1, 1998, p. 19-53.

    Research output: Contribution to journalArticle

    Aizenstein, H, Hegedus, T, Hellerstein, L & Pitt, L 1998, 'Complexity theoretic hardness results for query learning', Computational Complexity, vol. 7, no. 1, pp. 19-53.
    Aizenstein H, Hegedus T, Hellerstein L, Pitt L. Complexity theoretic hardness results for query learning. Computational Complexity. 1998;7(1):19-53.
    Aizenstein, Howard ; Hegedus, Tibor ; Hellerstein, Lisa ; Pitt, Leonard. / Complexity theoretic hardness results for query learning. In: Computational Complexity. 1998 ; Vol. 7, No. 1. pp. 19-53.
    @article{26777487d55b44fd8627658e275acae8,
    title = "Complexity theoretic hardness results for query learning",
    abstract = "We investigate the complexity of learning for the well-studied model in which the learning algorithm may ask membership and equivalence queries. While complexity theoretic techniques have previously been used to prove hardness results in various learning models, these techniques typically are not strong enough to use when a learning algorithm may make membership queries. We develop a general technique for proving hardness results for learning with membership and equivalence queries (and for more general query models). We apply the technique to show that, assuming NP ≠ co-NP, no polynomial-time membership and (proper) equivalence query algorithms exist for exactly learning read-thrice DNF formulas, unions of k ≥ 3 halfspaces over the Boolean domain, or some other related classes. Our hardness results are representation dependent, and do not preclude the existence of representation independent algorithms. The general technique introduces the representation problem for a class F of representations (e.g., formulas), which is naturally associated with the learning problem for F. This problem is related to the structural question of how to characterize functions representable by formulas in F, and is a generalization of standard complexity problems such as SATISFIABILITY. While in general the representation problem is in ΣP2, we present a theorem demonstrating that for {"}reasonable{"} classes F, the existence of a polynomial-time membership and equivalence query algorithm for exactly learning F implies that the representation problem for F is in fact in co-NP. The theorem is applied to prove hardness results such as the ones mentioned above, by showing that the representation problem for specific classes of formulas is NP-hard.",
    keywords = "Complexity theory, Computational learning theory, Equivalence queries, Membership queries, Query learning, Read-thrice DNF, Threshold functions",
    author = "Howard Aizenstein and Tibor Hegedus and Lisa Hellerstein and Leonard Pitt",
    year = "1998",
    language = "English (US)",
    volume = "7",
    pages = "19--53",
    journal = "Computational Complexity",
    issn = "1016-3328",
    publisher = "Birkhauser Verlag Basel",
    number = "1",

    }

    TY - JOUR

    T1 - Complexity theoretic hardness results for query learning

    AU - Aizenstein, Howard

    AU - Hegedus, Tibor

    AU - Hellerstein, Lisa

    AU - Pitt, Leonard

    PY - 1998

    Y1 - 1998

    N2 - We investigate the complexity of learning for the well-studied model in which the learning algorithm may ask membership and equivalence queries. While complexity theoretic techniques have previously been used to prove hardness results in various learning models, these techniques typically are not strong enough to use when a learning algorithm may make membership queries. We develop a general technique for proving hardness results for learning with membership and equivalence queries (and for more general query models). We apply the technique to show that, assuming NP ≠ co-NP, no polynomial-time membership and (proper) equivalence query algorithms exist for exactly learning read-thrice DNF formulas, unions of k ≥ 3 halfspaces over the Boolean domain, or some other related classes. Our hardness results are representation dependent, and do not preclude the existence of representation independent algorithms. The general technique introduces the representation problem for a class F of representations (e.g., formulas), which is naturally associated with the learning problem for F. This problem is related to the structural question of how to characterize functions representable by formulas in F, and is a generalization of standard complexity problems such as SATISFIABILITY. While in general the representation problem is in ΣP2, we present a theorem demonstrating that for "reasonable" classes F, the existence of a polynomial-time membership and equivalence query algorithm for exactly learning F implies that the representation problem for F is in fact in co-NP. The theorem is applied to prove hardness results such as the ones mentioned above, by showing that the representation problem for specific classes of formulas is NP-hard.

    AB - We investigate the complexity of learning for the well-studied model in which the learning algorithm may ask membership and equivalence queries. While complexity theoretic techniques have previously been used to prove hardness results in various learning models, these techniques typically are not strong enough to use when a learning algorithm may make membership queries. We develop a general technique for proving hardness results for learning with membership and equivalence queries (and for more general query models). We apply the technique to show that, assuming NP ≠ co-NP, no polynomial-time membership and (proper) equivalence query algorithms exist for exactly learning read-thrice DNF formulas, unions of k ≥ 3 halfspaces over the Boolean domain, or some other related classes. Our hardness results are representation dependent, and do not preclude the existence of representation independent algorithms. The general technique introduces the representation problem for a class F of representations (e.g., formulas), which is naturally associated with the learning problem for F. This problem is related to the structural question of how to characterize functions representable by formulas in F, and is a generalization of standard complexity problems such as SATISFIABILITY. While in general the representation problem is in ΣP2, we present a theorem demonstrating that for "reasonable" classes F, the existence of a polynomial-time membership and equivalence query algorithm for exactly learning F implies that the representation problem for F is in fact in co-NP. The theorem is applied to prove hardness results such as the ones mentioned above, by showing that the representation problem for specific classes of formulas is NP-hard.

    KW - Complexity theory

    KW - Computational learning theory

    KW - Equivalence queries

    KW - Membership queries

    KW - Query learning

    KW - Read-thrice DNF

    KW - Threshold functions

    UR - http://www.scopus.com/inward/record.url?scp=0032361270&partnerID=8YFLogxK

    UR - http://www.scopus.com/inward/citedby.url?scp=0032361270&partnerID=8YFLogxK

    M3 - Article

    VL - 7

    SP - 19

    EP - 53

    JO - Computational Complexity

    JF - Computational Complexity

    SN - 1016-3328

    IS - 1

    ER -