Using the shapley value to analyze algorithm portfolios

Leyton Brown Kevin, Alexandre Fŕechette, Lars Kotthoff, Tomasz Michalak, Talal Rahwan, Holger H. Hoos

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Abstract

    Algorithms for NP-complete problems often have different strengths and weaknesses, and thus algorithm portfolios often outperform individual algorithms. It is surprisingly difficult to quantify a component algorithm's contribution to such a portfolio. Reporting a component's standalone performance wrongly rewards near-clones while penalizing algorithms that have small but distinct areas of strength. Measuring a component's marginal contribution to an existing portfolio is better, but penalizes sets of strongly correlated algorithms, thereby obscuring situations in which it is essential to have at least one algorithm from such a set. This paper argues for analyzing component algorithm contributions via a measure drawn from coalitional game theory-The Shapley value-And yields insight into a research community's progress over time. We conclude with an application of the analysis we advocate to SAT competitions, yielding novel insights into the behaviour of algorithm portfolios, their components, and the state of SAT solving technology.

    Original languageEnglish (US)
    Title of host publication30th AAAI Conference on Artificial Intelligence, AAAI 2016
    PublisherAAAI press
    Pages3397-3403
    Number of pages7
    ISBN (Electronic)9781577357605
    StatePublished - Jan 1 2016
    Event30th AAAI Conference on Artificial Intelligence, AAAI 2016 - Phoenix, United States
    Duration: Feb 12 2016Feb 17 2016

    Other

    Other30th AAAI Conference on Artificial Intelligence, AAAI 2016
    CountryUnited States
    CityPhoenix
    Period2/12/162/17/16

    Fingerprint

    Game theory
    Computational complexity

    ASJC Scopus subject areas

    • Artificial Intelligence

    Cite this

    Kevin, L. B., Fŕechette, A., Kotthoff, L., Michalak, T., Rahwan, T., & Hoos, H. H. (2016). Using the shapley value to analyze algorithm portfolios. In 30th AAAI Conference on Artificial Intelligence, AAAI 2016 (pp. 3397-3403). AAAI press.

    Using the shapley value to analyze algorithm portfolios. / Kevin, Leyton Brown; Fŕechette, Alexandre; Kotthoff, Lars; Michalak, Tomasz; Rahwan, Talal; Hoos, Holger H.

    30th AAAI Conference on Artificial Intelligence, AAAI 2016. AAAI press, 2016. p. 3397-3403.

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Kevin, LB, Fŕechette, A, Kotthoff, L, Michalak, T, Rahwan, T & Hoos, HH 2016, Using the shapley value to analyze algorithm portfolios. in 30th AAAI Conference on Artificial Intelligence, AAAI 2016. AAAI press, pp. 3397-3403, 30th AAAI Conference on Artificial Intelligence, AAAI 2016, Phoenix, United States, 2/12/16.
    Kevin LB, Fŕechette A, Kotthoff L, Michalak T, Rahwan T, Hoos HH. Using the shapley value to analyze algorithm portfolios. In 30th AAAI Conference on Artificial Intelligence, AAAI 2016. AAAI press. 2016. p. 3397-3403
    Kevin, Leyton Brown ; Fŕechette, Alexandre ; Kotthoff, Lars ; Michalak, Tomasz ; Rahwan, Talal ; Hoos, Holger H. / Using the shapley value to analyze algorithm portfolios. 30th AAAI Conference on Artificial Intelligence, AAAI 2016. AAAI press, 2016. pp. 3397-3403
    @inproceedings{6c7147a4bd4540fea94ff73505160214,
    title = "Using the shapley value to analyze algorithm portfolios",
    abstract = "Algorithms for NP-complete problems often have different strengths and weaknesses, and thus algorithm portfolios often outperform individual algorithms. It is surprisingly difficult to quantify a component algorithm's contribution to such a portfolio. Reporting a component's standalone performance wrongly rewards near-clones while penalizing algorithms that have small but distinct areas of strength. Measuring a component's marginal contribution to an existing portfolio is better, but penalizes sets of strongly correlated algorithms, thereby obscuring situations in which it is essential to have at least one algorithm from such a set. This paper argues for analyzing component algorithm contributions via a measure drawn from coalitional game theory-The Shapley value-And yields insight into a research community's progress over time. We conclude with an application of the analysis we advocate to SAT competitions, yielding novel insights into the behaviour of algorithm portfolios, their components, and the state of SAT solving technology.",
    author = "Kevin, {Leyton Brown} and Alexandre Fŕechette and Lars Kotthoff and Tomasz Michalak and Talal Rahwan and Hoos, {Holger H.}",
    year = "2016",
    month = "1",
    day = "1",
    language = "English (US)",
    pages = "3397--3403",
    booktitle = "30th AAAI Conference on Artificial Intelligence, AAAI 2016",
    publisher = "AAAI press",

    }

    TY - GEN

    T1 - Using the shapley value to analyze algorithm portfolios

    AU - Kevin, Leyton Brown

    AU - Fŕechette, Alexandre

    AU - Kotthoff, Lars

    AU - Michalak, Tomasz

    AU - Rahwan, Talal

    AU - Hoos, Holger H.

    PY - 2016/1/1

    Y1 - 2016/1/1

    N2 - Algorithms for NP-complete problems often have different strengths and weaknesses, and thus algorithm portfolios often outperform individual algorithms. It is surprisingly difficult to quantify a component algorithm's contribution to such a portfolio. Reporting a component's standalone performance wrongly rewards near-clones while penalizing algorithms that have small but distinct areas of strength. Measuring a component's marginal contribution to an existing portfolio is better, but penalizes sets of strongly correlated algorithms, thereby obscuring situations in which it is essential to have at least one algorithm from such a set. This paper argues for analyzing component algorithm contributions via a measure drawn from coalitional game theory-The Shapley value-And yields insight into a research community's progress over time. We conclude with an application of the analysis we advocate to SAT competitions, yielding novel insights into the behaviour of algorithm portfolios, their components, and the state of SAT solving technology.

    AB - Algorithms for NP-complete problems often have different strengths and weaknesses, and thus algorithm portfolios often outperform individual algorithms. It is surprisingly difficult to quantify a component algorithm's contribution to such a portfolio. Reporting a component's standalone performance wrongly rewards near-clones while penalizing algorithms that have small but distinct areas of strength. Measuring a component's marginal contribution to an existing portfolio is better, but penalizes sets of strongly correlated algorithms, thereby obscuring situations in which it is essential to have at least one algorithm from such a set. This paper argues for analyzing component algorithm contributions via a measure drawn from coalitional game theory-The Shapley value-And yields insight into a research community's progress over time. We conclude with an application of the analysis we advocate to SAT competitions, yielding novel insights into the behaviour of algorithm portfolios, their components, and the state of SAT solving technology.

    UR - http://www.scopus.com/inward/record.url?scp=85007195951&partnerID=8YFLogxK

    UR - http://www.scopus.com/inward/citedby.url?scp=85007195951&partnerID=8YFLogxK

    M3 - Conference contribution

    SP - 3397

    EP - 3403

    BT - 30th AAAI Conference on Artificial Intelligence, AAAI 2016

    PB - AAAI press

    ER -