Fishing, commitment, and communication

A proposal for comprehensive nonbinding research registration

Macartan Humphreys, Raul Sanchez de la sierra, Peter van der Windt

    Research output: Contribution to journalReview article

    Abstract

    Social scientists generally enjoy substantial latitude in selecting measures and models for hypothesis testing. Coupled with publication and related biases, this latitude raises the concern that researchers may intentionally or unintentionally select models that yield positive findings, leading to an unreliable body of published research. To combat this "fishing" problem in medical studies, leading journals now require preregistration of designs that emphasize the prior identification of dependent and independent variables. However, we demonstrate here that even with this level of advanced specification, the scope for fishing is considerable when there is latitude over selection of covariates, subgroups, and other elements of an analysis plan. These concerns could be addressed through the use of a form of comprehensive registration. We experiment with such an approach in the context of an ongoing field experiment for which we drafted a complete "mock report" of findings using fake data on treatment assignment. We describe the advantages and disadvantages of this form of registration and propose that a comprehensive but nonbinding approach be adopted as a first step to combat fishing by social scientists. Likely effects of comprehensive but nonbinding registration are discussed, the principal advantage being communication rather than commitment, in particular that it generates a clear distinction between exploratory analyses and genuine tests.

    Original languageEnglish (US)
    Article numbermps021
    Pages (from-to)1-20
    Number of pages20
    JournalPolitical Analysis
    Volume21
    Issue number1
    DOIs
    StatePublished - Jan 1 2013

    Fingerprint

    commitment
    social scientist
    communication
    hypothesis testing
    experiment
    trend

    ASJC Scopus subject areas

    • Sociology and Political Science

    Cite this

    Fishing, commitment, and communication : A proposal for comprehensive nonbinding research registration. / Humphreys, Macartan; Sanchez de la sierra, Raul; van der Windt, Peter.

    In: Political Analysis, Vol. 21, No. 1, mps021, 01.01.2013, p. 1-20.

    Research output: Contribution to journalReview article

    Humphreys, M, Sanchez de la sierra, R & van der Windt, P 2013, 'Fishing, commitment, and communication: A proposal for comprehensive nonbinding research registration', Political Analysis, vol. 21, no. 1, mps021, pp. 1-20. https://doi.org/10.1093/pan/mps021
    Humphreys, Macartan ; Sanchez de la sierra, Raul ; van der Windt, Peter. / Fishing, commitment, and communication : A proposal for comprehensive nonbinding research registration. In: Political Analysis. 2013 ; Vol. 21, No. 1. pp. 1-20.
    @article{c08f0d38072745c5863f56f9556cdd13,
    title = "Fishing, commitment, and communication: A proposal for comprehensive nonbinding research registration",
    abstract = "Social scientists generally enjoy substantial latitude in selecting measures and models for hypothesis testing. Coupled with publication and related biases, this latitude raises the concern that researchers may intentionally or unintentionally select models that yield positive findings, leading to an unreliable body of published research. To combat this {"}fishing{"} problem in medical studies, leading journals now require preregistration of designs that emphasize the prior identification of dependent and independent variables. However, we demonstrate here that even with this level of advanced specification, the scope for fishing is considerable when there is latitude over selection of covariates, subgroups, and other elements of an analysis plan. These concerns could be addressed through the use of a form of comprehensive registration. We experiment with such an approach in the context of an ongoing field experiment for which we drafted a complete {"}mock report{"} of findings using fake data on treatment assignment. We describe the advantages and disadvantages of this form of registration and propose that a comprehensive but nonbinding approach be adopted as a first step to combat fishing by social scientists. Likely effects of comprehensive but nonbinding registration are discussed, the principal advantage being communication rather than commitment, in particular that it generates a clear distinction between exploratory analyses and genuine tests.",
    author = "Macartan Humphreys and {Sanchez de la sierra}, Raul and {van der Windt}, Peter",
    year = "2013",
    month = "1",
    day = "1",
    doi = "10.1093/pan/mps021",
    language = "English (US)",
    volume = "21",
    pages = "1--20",
    journal = "Political Analysis",
    issn = "1047-1987",
    publisher = "Oxford University Press",
    number = "1",

    }

    TY - JOUR

    T1 - Fishing, commitment, and communication

    T2 - A proposal for comprehensive nonbinding research registration

    AU - Humphreys, Macartan

    AU - Sanchez de la sierra, Raul

    AU - van der Windt, Peter

    PY - 2013/1/1

    Y1 - 2013/1/1

    N2 - Social scientists generally enjoy substantial latitude in selecting measures and models for hypothesis testing. Coupled with publication and related biases, this latitude raises the concern that researchers may intentionally or unintentionally select models that yield positive findings, leading to an unreliable body of published research. To combat this "fishing" problem in medical studies, leading journals now require preregistration of designs that emphasize the prior identification of dependent and independent variables. However, we demonstrate here that even with this level of advanced specification, the scope for fishing is considerable when there is latitude over selection of covariates, subgroups, and other elements of an analysis plan. These concerns could be addressed through the use of a form of comprehensive registration. We experiment with such an approach in the context of an ongoing field experiment for which we drafted a complete "mock report" of findings using fake data on treatment assignment. We describe the advantages and disadvantages of this form of registration and propose that a comprehensive but nonbinding approach be adopted as a first step to combat fishing by social scientists. Likely effects of comprehensive but nonbinding registration are discussed, the principal advantage being communication rather than commitment, in particular that it generates a clear distinction between exploratory analyses and genuine tests.

    AB - Social scientists generally enjoy substantial latitude in selecting measures and models for hypothesis testing. Coupled with publication and related biases, this latitude raises the concern that researchers may intentionally or unintentionally select models that yield positive findings, leading to an unreliable body of published research. To combat this "fishing" problem in medical studies, leading journals now require preregistration of designs that emphasize the prior identification of dependent and independent variables. However, we demonstrate here that even with this level of advanced specification, the scope for fishing is considerable when there is latitude over selection of covariates, subgroups, and other elements of an analysis plan. These concerns could be addressed through the use of a form of comprehensive registration. We experiment with such an approach in the context of an ongoing field experiment for which we drafted a complete "mock report" of findings using fake data on treatment assignment. We describe the advantages and disadvantages of this form of registration and propose that a comprehensive but nonbinding approach be adopted as a first step to combat fishing by social scientists. Likely effects of comprehensive but nonbinding registration are discussed, the principal advantage being communication rather than commitment, in particular that it generates a clear distinction between exploratory analyses and genuine tests.

    UR - http://www.scopus.com/inward/record.url?scp=84872389242&partnerID=8YFLogxK

    UR - http://www.scopus.com/inward/citedby.url?scp=84872389242&partnerID=8YFLogxK

    U2 - 10.1093/pan/mps021

    DO - 10.1093/pan/mps021

    M3 - Review article

    VL - 21

    SP - 1

    EP - 20

    JO - Political Analysis

    JF - Political Analysis

    SN - 1047-1987

    IS - 1

    M1 - mps021

    ER -