Offline evaluation of ranking policies with click models

Shuai Li, Shanmugavelayutham Muthukrishnan, Yasin Abbasi-Yadkori, Vishwa Vinay, Branislav Kveton, Zheng Wen

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Abstract

    Many web systems rank and present a list of items to users, from recommender systems to search and advertising. An important problem in practice is to evaluate new ranking policies offline and optimize them before they are deployed. We address this problem by proposing evaluation algorithms for estimating the expected number of clicks on ranked lists from historical logged data. The existing algorithms are not guaranteed to be statistically efficient in our problem because the number of recommended lists can grow exponentially with their length. To overcome this challenge, we use models of user interaction with the list of items, the so-called click models, to construct estimators that learn statistically efficiently. We analyze our estimators and prove that they are more efficient than the estimators that do not use the structure of the click model, under the assumption that the click model holds. We evaluate our estimators in a series of experiments on a real-world dataset and show that they consistently outperform prior estimators.

    Original languageEnglish (US)
    Title of host publicationKDD 2018 - Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining
    PublisherAssociation for Computing Machinery
    Pages1685-1694
    Number of pages10
    ISBN (Print)9781450355520
    DOIs
    StatePublished - Jul 19 2018
    Event24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2018 - London, United Kingdom
    Duration: Aug 19 2018Aug 23 2018

    Publication series

    NameProceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

    Other

    Other24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2018
    CountryUnited Kingdom
    CityLondon
    Period8/19/188/23/18

    Fingerprint

    Recommender systems
    Marketing
    Experiments

    Keywords

    • Click models
    • Importance sampling
    • Offline evaluation
    • Ranking

    ASJC Scopus subject areas

    • Software
    • Information Systems

    Cite this

    Li, S., Muthukrishnan, S., Abbasi-Yadkori, Y., Vinay, V., Kveton, B., & Wen, Z. (2018). Offline evaluation of ranking policies with click models. In KDD 2018 - Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 1685-1694). (Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining). Association for Computing Machinery. https://doi.org/10.1145/3219819.3220028

    Offline evaluation of ranking policies with click models. / Li, Shuai; Muthukrishnan, Shanmugavelayutham; Abbasi-Yadkori, Yasin; Vinay, Vishwa; Kveton, Branislav; Wen, Zheng.

    KDD 2018 - Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Association for Computing Machinery, 2018. p. 1685-1694 (Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining).

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Li, S, Muthukrishnan, S, Abbasi-Yadkori, Y, Vinay, V, Kveton, B & Wen, Z 2018, Offline evaluation of ranking policies with click models. in KDD 2018 - Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Association for Computing Machinery, pp. 1685-1694, 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2018, London, United Kingdom, 8/19/18. https://doi.org/10.1145/3219819.3220028
    Li S, Muthukrishnan S, Abbasi-Yadkori Y, Vinay V, Kveton B, Wen Z. Offline evaluation of ranking policies with click models. In KDD 2018 - Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Association for Computing Machinery. 2018. p. 1685-1694. (Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining). https://doi.org/10.1145/3219819.3220028
    Li, Shuai ; Muthukrishnan, Shanmugavelayutham ; Abbasi-Yadkori, Yasin ; Vinay, Vishwa ; Kveton, Branislav ; Wen, Zheng. / Offline evaluation of ranking policies with click models. KDD 2018 - Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Association for Computing Machinery, 2018. pp. 1685-1694 (Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining).
    @inproceedings{4ea029e379d245c088dd3a59a271f926,
    title = "Offline evaluation of ranking policies with click models",
    abstract = "Many web systems rank and present a list of items to users, from recommender systems to search and advertising. An important problem in practice is to evaluate new ranking policies offline and optimize them before they are deployed. We address this problem by proposing evaluation algorithms for estimating the expected number of clicks on ranked lists from historical logged data. The existing algorithms are not guaranteed to be statistically efficient in our problem because the number of recommended lists can grow exponentially with their length. To overcome this challenge, we use models of user interaction with the list of items, the so-called click models, to construct estimators that learn statistically efficiently. We analyze our estimators and prove that they are more efficient than the estimators that do not use the structure of the click model, under the assumption that the click model holds. We evaluate our estimators in a series of experiments on a real-world dataset and show that they consistently outperform prior estimators.",
    keywords = "Click models, Importance sampling, Offline evaluation, Ranking",
    author = "Shuai Li and Shanmugavelayutham Muthukrishnan and Yasin Abbasi-Yadkori and Vishwa Vinay and Branislav Kveton and Zheng Wen",
    year = "2018",
    month = "7",
    day = "19",
    doi = "10.1145/3219819.3220028",
    language = "English (US)",
    isbn = "9781450355520",
    series = "Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining",
    publisher = "Association for Computing Machinery",
    pages = "1685--1694",
    booktitle = "KDD 2018 - Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining",

    }

    TY - GEN

    T1 - Offline evaluation of ranking policies with click models

    AU - Li, Shuai

    AU - Muthukrishnan, Shanmugavelayutham

    AU - Abbasi-Yadkori, Yasin

    AU - Vinay, Vishwa

    AU - Kveton, Branislav

    AU - Wen, Zheng

    PY - 2018/7/19

    Y1 - 2018/7/19

    N2 - Many web systems rank and present a list of items to users, from recommender systems to search and advertising. An important problem in practice is to evaluate new ranking policies offline and optimize them before they are deployed. We address this problem by proposing evaluation algorithms for estimating the expected number of clicks on ranked lists from historical logged data. The existing algorithms are not guaranteed to be statistically efficient in our problem because the number of recommended lists can grow exponentially with their length. To overcome this challenge, we use models of user interaction with the list of items, the so-called click models, to construct estimators that learn statistically efficiently. We analyze our estimators and prove that they are more efficient than the estimators that do not use the structure of the click model, under the assumption that the click model holds. We evaluate our estimators in a series of experiments on a real-world dataset and show that they consistently outperform prior estimators.

    AB - Many web systems rank and present a list of items to users, from recommender systems to search and advertising. An important problem in practice is to evaluate new ranking policies offline and optimize them before they are deployed. We address this problem by proposing evaluation algorithms for estimating the expected number of clicks on ranked lists from historical logged data. The existing algorithms are not guaranteed to be statistically efficient in our problem because the number of recommended lists can grow exponentially with their length. To overcome this challenge, we use models of user interaction with the list of items, the so-called click models, to construct estimators that learn statistically efficiently. We analyze our estimators and prove that they are more efficient than the estimators that do not use the structure of the click model, under the assumption that the click model holds. We evaluate our estimators in a series of experiments on a real-world dataset and show that they consistently outperform prior estimators.

    KW - Click models

    KW - Importance sampling

    KW - Offline evaluation

    KW - Ranking

    UR - http://www.scopus.com/inward/record.url?scp=85051464560&partnerID=8YFLogxK

    UR - http://www.scopus.com/inward/citedby.url?scp=85051464560&partnerID=8YFLogxK

    U2 - 10.1145/3219819.3220028

    DO - 10.1145/3219819.3220028

    M3 - Conference contribution

    AN - SCOPUS:85051464560

    SN - 9781450355520

    T3 - Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

    SP - 1685

    EP - 1694

    BT - KDD 2018 - Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

    PB - Association for Computing Machinery

    ER -