Fishing, commitment, and communication: A proposal for comprehensive nonbinding research registration

Macartan Humphreys, Raul Sanchez de la sierra, Peter van der Windt

Research output: Contribution to journalReview article

Abstract

Social scientists generally enjoy substantial latitude in selecting measures and models for hypothesis testing. Coupled with publication and related biases, this latitude raises the concern that researchers may intentionally or unintentionally select models that yield positive findings, leading to an unreliable body of published research. To combat this "fishing" problem in medical studies, leading journals now require preregistration of designs that emphasize the prior identification of dependent and independent variables. However, we demonstrate here that even with this level of advanced specification, the scope for fishing is considerable when there is latitude over selection of covariates, subgroups, and other elements of an analysis plan. These concerns could be addressed through the use of a form of comprehensive registration. We experiment with such an approach in the context of an ongoing field experiment for which we drafted a complete "mock report" of findings using fake data on treatment assignment. We describe the advantages and disadvantages of this form of registration and propose that a comprehensive but nonbinding approach be adopted as a first step to combat fishing by social scientists. Likely effects of comprehensive but nonbinding registration are discussed, the principal advantage being communication rather than commitment, in particular that it generates a clear distinction between exploratory analyses and genuine tests.

Original languageEnglish (US)
Article numbermps021
Pages (from-to)1-20
Number of pages20
JournalPolitical Analysis
Volume21
Issue number1
DOIs
StatePublished - Jan 1 2013

Fingerprint

commitment
social scientist
communication
hypothesis testing
experiment
trend

ASJC Scopus subject areas

  • Sociology and Political Science

Cite this

Fishing, commitment, and communication : A proposal for comprehensive nonbinding research registration. / Humphreys, Macartan; Sanchez de la sierra, Raul; van der Windt, Peter.

In: Political Analysis, Vol. 21, No. 1, mps021, 01.01.2013, p. 1-20.

Research output: Contribution to journalReview article

@article{c08f0d38072745c5863f56f9556cdd13,
title = "Fishing, commitment, and communication: A proposal for comprehensive nonbinding research registration",
abstract = "Social scientists generally enjoy substantial latitude in selecting measures and models for hypothesis testing. Coupled with publication and related biases, this latitude raises the concern that researchers may intentionally or unintentionally select models that yield positive findings, leading to an unreliable body of published research. To combat this {"}fishing{"} problem in medical studies, leading journals now require preregistration of designs that emphasize the prior identification of dependent and independent variables. However, we demonstrate here that even with this level of advanced specification, the scope for fishing is considerable when there is latitude over selection of covariates, subgroups, and other elements of an analysis plan. These concerns could be addressed through the use of a form of comprehensive registration. We experiment with such an approach in the context of an ongoing field experiment for which we drafted a complete {"}mock report{"} of findings using fake data on treatment assignment. We describe the advantages and disadvantages of this form of registration and propose that a comprehensive but nonbinding approach be adopted as a first step to combat fishing by social scientists. Likely effects of comprehensive but nonbinding registration are discussed, the principal advantage being communication rather than commitment, in particular that it generates a clear distinction between exploratory analyses and genuine tests.",
author = "Macartan Humphreys and {Sanchez de la sierra}, Raul and {van der Windt}, Peter",
year = "2013",
month = "1",
day = "1",
doi = "10.1093/pan/mps021",
language = "English (US)",
volume = "21",
pages = "1--20",
journal = "Political Analysis",
issn = "1047-1987",
publisher = "Oxford University Press",
number = "1",

}

TY - JOUR

T1 - Fishing, commitment, and communication

T2 - A proposal for comprehensive nonbinding research registration

AU - Humphreys, Macartan

AU - Sanchez de la sierra, Raul

AU - van der Windt, Peter

PY - 2013/1/1

Y1 - 2013/1/1

N2 - Social scientists generally enjoy substantial latitude in selecting measures and models for hypothesis testing. Coupled with publication and related biases, this latitude raises the concern that researchers may intentionally or unintentionally select models that yield positive findings, leading to an unreliable body of published research. To combat this "fishing" problem in medical studies, leading journals now require preregistration of designs that emphasize the prior identification of dependent and independent variables. However, we demonstrate here that even with this level of advanced specification, the scope for fishing is considerable when there is latitude over selection of covariates, subgroups, and other elements of an analysis plan. These concerns could be addressed through the use of a form of comprehensive registration. We experiment with such an approach in the context of an ongoing field experiment for which we drafted a complete "mock report" of findings using fake data on treatment assignment. We describe the advantages and disadvantages of this form of registration and propose that a comprehensive but nonbinding approach be adopted as a first step to combat fishing by social scientists. Likely effects of comprehensive but nonbinding registration are discussed, the principal advantage being communication rather than commitment, in particular that it generates a clear distinction between exploratory analyses and genuine tests.

AB - Social scientists generally enjoy substantial latitude in selecting measures and models for hypothesis testing. Coupled with publication and related biases, this latitude raises the concern that researchers may intentionally or unintentionally select models that yield positive findings, leading to an unreliable body of published research. To combat this "fishing" problem in medical studies, leading journals now require preregistration of designs that emphasize the prior identification of dependent and independent variables. However, we demonstrate here that even with this level of advanced specification, the scope for fishing is considerable when there is latitude over selection of covariates, subgroups, and other elements of an analysis plan. These concerns could be addressed through the use of a form of comprehensive registration. We experiment with such an approach in the context of an ongoing field experiment for which we drafted a complete "mock report" of findings using fake data on treatment assignment. We describe the advantages and disadvantages of this form of registration and propose that a comprehensive but nonbinding approach be adopted as a first step to combat fishing by social scientists. Likely effects of comprehensive but nonbinding registration are discussed, the principal advantage being communication rather than commitment, in particular that it generates a clear distinction between exploratory analyses and genuine tests.

UR - http://www.scopus.com/inward/record.url?scp=84872389242&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84872389242&partnerID=8YFLogxK

U2 - 10.1093/pan/mps021

DO - 10.1093/pan/mps021

M3 - Review article

AN - SCOPUS:84872389242

VL - 21

SP - 1

EP - 20

JO - Political Analysis

JF - Political Analysis

SN - 1047-1987

IS - 1

M1 - mps021

ER -