Using the shapley value to analyze algorithm portfolios

Leyton Brown Kevin, Alexandre Fŕechette, Lars Kotthoff, Tomasz Michalak, Talal Rahwan, Holger H. Hoos

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Algorithms for NP-complete problems often have different strengths and weaknesses, and thus algorithm portfolios often outperform individual algorithms. It is surprisingly difficult to quantify a component algorithm's contribution to such a portfolio. Reporting a component's standalone performance wrongly rewards near-clones while penalizing algorithms that have small but distinct areas of strength. Measuring a component's marginal contribution to an existing portfolio is better, but penalizes sets of strongly correlated algorithms, thereby obscuring situations in which it is essential to have at least one algorithm from such a set. This paper argues for analyzing component algorithm contributions via a measure drawn from coalitional game theory-The Shapley value-And yields insight into a research community's progress over time. We conclude with an application of the analysis we advocate to SAT competitions, yielding novel insights into the behaviour of algorithm portfolios, their components, and the state of SAT solving technology.

Original languageEnglish (US)
Title of host publication30th AAAI Conference on Artificial Intelligence, AAAI 2016
PublisherAAAI press
Pages3397-3403
Number of pages7
ISBN (Electronic)9781577357605
StatePublished - Jan 1 2016
Event30th AAAI Conference on Artificial Intelligence, AAAI 2016 - Phoenix, United States
Duration: Feb 12 2016Feb 17 2016

Other

Other30th AAAI Conference on Artificial Intelligence, AAAI 2016
CountryUnited States
CityPhoenix
Period2/12/162/17/16

Fingerprint

Game theory
Computational complexity

ASJC Scopus subject areas

  • Artificial Intelligence

Cite this

Kevin, L. B., Fŕechette, A., Kotthoff, L., Michalak, T., Rahwan, T., & Hoos, H. H. (2016). Using the shapley value to analyze algorithm portfolios. In 30th AAAI Conference on Artificial Intelligence, AAAI 2016 (pp. 3397-3403). AAAI press.

Using the shapley value to analyze algorithm portfolios. / Kevin, Leyton Brown; Fŕechette, Alexandre; Kotthoff, Lars; Michalak, Tomasz; Rahwan, Talal; Hoos, Holger H.

30th AAAI Conference on Artificial Intelligence, AAAI 2016. AAAI press, 2016. p. 3397-3403.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Kevin, LB, Fŕechette, A, Kotthoff, L, Michalak, T, Rahwan, T & Hoos, HH 2016, Using the shapley value to analyze algorithm portfolios. in 30th AAAI Conference on Artificial Intelligence, AAAI 2016. AAAI press, pp. 3397-3403, 30th AAAI Conference on Artificial Intelligence, AAAI 2016, Phoenix, United States, 2/12/16.
Kevin LB, Fŕechette A, Kotthoff L, Michalak T, Rahwan T, Hoos HH. Using the shapley value to analyze algorithm portfolios. In 30th AAAI Conference on Artificial Intelligence, AAAI 2016. AAAI press. 2016. p. 3397-3403
Kevin, Leyton Brown ; Fŕechette, Alexandre ; Kotthoff, Lars ; Michalak, Tomasz ; Rahwan, Talal ; Hoos, Holger H. / Using the shapley value to analyze algorithm portfolios. 30th AAAI Conference on Artificial Intelligence, AAAI 2016. AAAI press, 2016. pp. 3397-3403
@inproceedings{525638e89d6c4628b82e76f912f472f0,
title = "Using the shapley value to analyze algorithm portfolios",
abstract = "Algorithms for NP-complete problems often have different strengths and weaknesses, and thus algorithm portfolios often outperform individual algorithms. It is surprisingly difficult to quantify a component algorithm's contribution to such a portfolio. Reporting a component's standalone performance wrongly rewards near-clones while penalizing algorithms that have small but distinct areas of strength. Measuring a component's marginal contribution to an existing portfolio is better, but penalizes sets of strongly correlated algorithms, thereby obscuring situations in which it is essential to have at least one algorithm from such a set. This paper argues for analyzing component algorithm contributions via a measure drawn from coalitional game theory-The Shapley value-And yields insight into a research community's progress over time. We conclude with an application of the analysis we advocate to SAT competitions, yielding novel insights into the behaviour of algorithm portfolios, their components, and the state of SAT solving technology.",
author = "Kevin, {Leyton Brown} and Alexandre Fŕechette and Lars Kotthoff and Tomasz Michalak and Talal Rahwan and Hoos, {Holger H.}",
year = "2016",
month = "1",
day = "1",
language = "English (US)",
pages = "3397--3403",
booktitle = "30th AAAI Conference on Artificial Intelligence, AAAI 2016",
publisher = "AAAI press",

}

TY - GEN

T1 - Using the shapley value to analyze algorithm portfolios

AU - Kevin, Leyton Brown

AU - Fŕechette, Alexandre

AU - Kotthoff, Lars

AU - Michalak, Tomasz

AU - Rahwan, Talal

AU - Hoos, Holger H.

PY - 2016/1/1

Y1 - 2016/1/1

N2 - Algorithms for NP-complete problems often have different strengths and weaknesses, and thus algorithm portfolios often outperform individual algorithms. It is surprisingly difficult to quantify a component algorithm's contribution to such a portfolio. Reporting a component's standalone performance wrongly rewards near-clones while penalizing algorithms that have small but distinct areas of strength. Measuring a component's marginal contribution to an existing portfolio is better, but penalizes sets of strongly correlated algorithms, thereby obscuring situations in which it is essential to have at least one algorithm from such a set. This paper argues for analyzing component algorithm contributions via a measure drawn from coalitional game theory-The Shapley value-And yields insight into a research community's progress over time. We conclude with an application of the analysis we advocate to SAT competitions, yielding novel insights into the behaviour of algorithm portfolios, their components, and the state of SAT solving technology.

AB - Algorithms for NP-complete problems often have different strengths and weaknesses, and thus algorithm portfolios often outperform individual algorithms. It is surprisingly difficult to quantify a component algorithm's contribution to such a portfolio. Reporting a component's standalone performance wrongly rewards near-clones while penalizing algorithms that have small but distinct areas of strength. Measuring a component's marginal contribution to an existing portfolio is better, but penalizes sets of strongly correlated algorithms, thereby obscuring situations in which it is essential to have at least one algorithm from such a set. This paper argues for analyzing component algorithm contributions via a measure drawn from coalitional game theory-The Shapley value-And yields insight into a research community's progress over time. We conclude with an application of the analysis we advocate to SAT competitions, yielding novel insights into the behaviour of algorithm portfolios, their components, and the state of SAT solving technology.

UR - http://www.scopus.com/inward/record.url?scp=85007195951&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85007195951&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85007195951

SP - 3397

EP - 3403

BT - 30th AAAI Conference on Artificial Intelligence, AAAI 2016

PB - AAAI press

ER -