Classification Accuracy as a Substantive Quantity of Interest

Measuring Polarization in Westminster Systems

Andrew Peterson, Arthur Spirling

    Research output: Contribution to journalArticle

    Abstract

    Measuring the polarization of legislators and parties is a key step in understanding how politics develops over time. But in parliamentary systems-where ideological positions estimated from roll calls may not be informative-producing valid estimates is extremely challenging. We suggest a new measurement strategy that makes innovative use of the accuracy of machine classifiers, i.e., the number of correct predictions made as a proportion of all predictions. In our case, the labels are the party identifications of the members of parliament, predicted from their speeches along with some information on debate subjects. Intuitively, when the learner is able to discriminate members in the two main Westminster parties well, we claim we are in a period of high polarization. By contrast, when the classifier has low accuracy- A nd makes a relatively large number of mistakes in terms of allocating members to parties based on the data-we argue parliament is in an era of low polarization. This approach is fast and substantively valid, and we demonstrate its merits with simulations, and by comparing the estimates from 78 years of House of Commons speeches with qualitative and quantitative historical accounts of the same. As a headline finding, we note that contemporary British politics is approximately as polarized as it was in the mid-1960s-that is, in the middle of the postwar consensus. More broadly, we show that the technical performance of supervised learning algorithms can be directly informative about substantive matters in social science.

    Original languageEnglish (US)
    Pages (from-to)120-128
    Number of pages9
    JournalPolitical Analysis
    Volume26
    Issue number1
    DOIs
    StatePublished - Jan 1 2018

    Fingerprint

    polarization
    parliament
    politics
    social science
    simulation
    learning
    performance
    time

    Keywords

    • learning
    • polarization
    • Statistical analysis of texts

    ASJC Scopus subject areas

    • Sociology and Political Science
    • Political Science and International Relations

    Cite this

    Classification Accuracy as a Substantive Quantity of Interest : Measuring Polarization in Westminster Systems. / Peterson, Andrew; Spirling, Arthur.

    In: Political Analysis, Vol. 26, No. 1, 01.01.2018, p. 120-128.

    Research output: Contribution to journalArticle

    @article{8c239cb4de0147eab6b7cceb1031b61a,
    title = "Classification Accuracy as a Substantive Quantity of Interest: Measuring Polarization in Westminster Systems",
    abstract = "Measuring the polarization of legislators and parties is a key step in understanding how politics develops over time. But in parliamentary systems-where ideological positions estimated from roll calls may not be informative-producing valid estimates is extremely challenging. We suggest a new measurement strategy that makes innovative use of the accuracy of machine classifiers, i.e., the number of correct predictions made as a proportion of all predictions. In our case, the labels are the party identifications of the members of parliament, predicted from their speeches along with some information on debate subjects. Intuitively, when the learner is able to discriminate members in the two main Westminster parties well, we claim we are in a period of high polarization. By contrast, when the classifier has low accuracy- A nd makes a relatively large number of mistakes in terms of allocating members to parties based on the data-we argue parliament is in an era of low polarization. This approach is fast and substantively valid, and we demonstrate its merits with simulations, and by comparing the estimates from 78 years of House of Commons speeches with qualitative and quantitative historical accounts of the same. As a headline finding, we note that contemporary British politics is approximately as polarized as it was in the mid-1960s-that is, in the middle of the postwar consensus. More broadly, we show that the technical performance of supervised learning algorithms can be directly informative about substantive matters in social science.",
    keywords = "learning, polarization, Statistical analysis of texts",
    author = "Andrew Peterson and Arthur Spirling",
    year = "2018",
    month = "1",
    day = "1",
    doi = "10.1017/pan.2017.39",
    language = "English (US)",
    volume = "26",
    pages = "120--128",
    journal = "Political Analysis",
    issn = "1047-1987",
    publisher = "Oxford University Press",
    number = "1",

    }

    TY - JOUR

    T1 - Classification Accuracy as a Substantive Quantity of Interest

    T2 - Measuring Polarization in Westminster Systems

    AU - Peterson, Andrew

    AU - Spirling, Arthur

    PY - 2018/1/1

    Y1 - 2018/1/1

    N2 - Measuring the polarization of legislators and parties is a key step in understanding how politics develops over time. But in parliamentary systems-where ideological positions estimated from roll calls may not be informative-producing valid estimates is extremely challenging. We suggest a new measurement strategy that makes innovative use of the accuracy of machine classifiers, i.e., the number of correct predictions made as a proportion of all predictions. In our case, the labels are the party identifications of the members of parliament, predicted from their speeches along with some information on debate subjects. Intuitively, when the learner is able to discriminate members in the two main Westminster parties well, we claim we are in a period of high polarization. By contrast, when the classifier has low accuracy- A nd makes a relatively large number of mistakes in terms of allocating members to parties based on the data-we argue parliament is in an era of low polarization. This approach is fast and substantively valid, and we demonstrate its merits with simulations, and by comparing the estimates from 78 years of House of Commons speeches with qualitative and quantitative historical accounts of the same. As a headline finding, we note that contemporary British politics is approximately as polarized as it was in the mid-1960s-that is, in the middle of the postwar consensus. More broadly, we show that the technical performance of supervised learning algorithms can be directly informative about substantive matters in social science.

    AB - Measuring the polarization of legislators and parties is a key step in understanding how politics develops over time. But in parliamentary systems-where ideological positions estimated from roll calls may not be informative-producing valid estimates is extremely challenging. We suggest a new measurement strategy that makes innovative use of the accuracy of machine classifiers, i.e., the number of correct predictions made as a proportion of all predictions. In our case, the labels are the party identifications of the members of parliament, predicted from their speeches along with some information on debate subjects. Intuitively, when the learner is able to discriminate members in the two main Westminster parties well, we claim we are in a period of high polarization. By contrast, when the classifier has low accuracy- A nd makes a relatively large number of mistakes in terms of allocating members to parties based on the data-we argue parliament is in an era of low polarization. This approach is fast and substantively valid, and we demonstrate its merits with simulations, and by comparing the estimates from 78 years of House of Commons speeches with qualitative and quantitative historical accounts of the same. As a headline finding, we note that contemporary British politics is approximately as polarized as it was in the mid-1960s-that is, in the middle of the postwar consensus. More broadly, we show that the technical performance of supervised learning algorithms can be directly informative about substantive matters in social science.

    KW - learning

    KW - polarization

    KW - Statistical analysis of texts

    UR - http://www.scopus.com/inward/record.url?scp=85041232594&partnerID=8YFLogxK

    UR - http://www.scopus.com/inward/citedby.url?scp=85041232594&partnerID=8YFLogxK

    U2 - 10.1017/pan.2017.39

    DO - 10.1017/pan.2017.39

    M3 - Article

    VL - 26

    SP - 120

    EP - 128

    JO - Political Analysis

    JF - Political Analysis

    SN - 1047-1987

    IS - 1

    ER -