Contextual sensitivity in scientific reproducibility

Jay Van Bavel, Peter Mende-Siedlecki, William J. Brady, Diego A. Reinero

Research output: Contribution to journalArticle

Abstract

In recent years, scientists have paid increasing attention to reproducibility. For example, the Reproducibility Project, a large-scale replication attempt of 100 studies published in top psychology journals found that only 39% could be unambiguously reproduced. There is a growing consensus among scientists that the lack of reproducibility in psychology and other fields stems from various methodological factors, including low statistical power, researcher's degrees of freedom, and an emphasis on publishing surprising positive results. However, there is a contentious debate about the extent to which failures to reproduce certain results might also reflect contextual differences (often termed "hiddenmoderators") between the original research and the replication attempt. Although psychologists have found extensive evidence that contextual factors alter behavior, some have argued that context is unlikely to influence the results of direct replications precisely because these studies use the same methods as those used in the original research. To help resolve this debate, we recoded the 100 original studies from the Reproducibility Project on the extent to which the research topic of each study was contextually sensitive. Results suggested that the contextual sensitivity of the research topic was associated with replication success, even after statistically adjusting for several methodological characteristics (e.g., statistical power, effect size). The association between contextual sensitivity and replication success did not differ across psychological subdisciplines. These results suggest that researchers, replicators, and consumers should bemindful of contextual factors that might influence a psychological process. We offer several guidelines for dealing with contextual sensitivity in reproducibility.

Original languageEnglish (US)
Pages (from-to)6454-6459
Number of pages6
JournalProceedings of the National Academy of Sciences of the United States of America
Volume113
Issue number23
DOIs
StatePublished - Jun 7 2016

Fingerprint

Psychology
Research
Research Personnel
Consensus
Guidelines

Keywords

  • Context
  • Meta-science
  • Psychology
  • Replication
  • Reproducibility

ASJC Scopus subject areas

  • General

Cite this

Contextual sensitivity in scientific reproducibility. / Van Bavel, Jay; Mende-Siedlecki, Peter; Brady, William J.; Reinero, Diego A.

In: Proceedings of the National Academy of Sciences of the United States of America, Vol. 113, No. 23, 07.06.2016, p. 6454-6459.

Research output: Contribution to journalArticle

Van Bavel, Jay ; Mende-Siedlecki, Peter ; Brady, William J. ; Reinero, Diego A. / Contextual sensitivity in scientific reproducibility. In: Proceedings of the National Academy of Sciences of the United States of America. 2016 ; Vol. 113, No. 23. pp. 6454-6459.
@article{c812f88b7e254126bcf9ee252f72f516,
title = "Contextual sensitivity in scientific reproducibility",
abstract = "In recent years, scientists have paid increasing attention to reproducibility. For example, the Reproducibility Project, a large-scale replication attempt of 100 studies published in top psychology journals found that only 39{\%} could be unambiguously reproduced. There is a growing consensus among scientists that the lack of reproducibility in psychology and other fields stems from various methodological factors, including low statistical power, researcher's degrees of freedom, and an emphasis on publishing surprising positive results. However, there is a contentious debate about the extent to which failures to reproduce certain results might also reflect contextual differences (often termed {"}hiddenmoderators{"}) between the original research and the replication attempt. Although psychologists have found extensive evidence that contextual factors alter behavior, some have argued that context is unlikely to influence the results of direct replications precisely because these studies use the same methods as those used in the original research. To help resolve this debate, we recoded the 100 original studies from the Reproducibility Project on the extent to which the research topic of each study was contextually sensitive. Results suggested that the contextual sensitivity of the research topic was associated with replication success, even after statistically adjusting for several methodological characteristics (e.g., statistical power, effect size). The association between contextual sensitivity and replication success did not differ across psychological subdisciplines. These results suggest that researchers, replicators, and consumers should bemindful of contextual factors that might influence a psychological process. We offer several guidelines for dealing with contextual sensitivity in reproducibility.",
keywords = "Context, Meta-science, Psychology, Replication, Reproducibility",
author = "{Van Bavel}, Jay and Peter Mende-Siedlecki and Brady, {William J.} and Reinero, {Diego A.}",
year = "2016",
month = "6",
day = "7",
doi = "10.1073/pnas.1521897113",
language = "English (US)",
volume = "113",
pages = "6454--6459",
journal = "Proceedings of the National Academy of Sciences of the United States of America",
issn = "0027-8424",
number = "23",

}

TY - JOUR

T1 - Contextual sensitivity in scientific reproducibility

AU - Van Bavel, Jay

AU - Mende-Siedlecki, Peter

AU - Brady, William J.

AU - Reinero, Diego A.

PY - 2016/6/7

Y1 - 2016/6/7

N2 - In recent years, scientists have paid increasing attention to reproducibility. For example, the Reproducibility Project, a large-scale replication attempt of 100 studies published in top psychology journals found that only 39% could be unambiguously reproduced. There is a growing consensus among scientists that the lack of reproducibility in psychology and other fields stems from various methodological factors, including low statistical power, researcher's degrees of freedom, and an emphasis on publishing surprising positive results. However, there is a contentious debate about the extent to which failures to reproduce certain results might also reflect contextual differences (often termed "hiddenmoderators") between the original research and the replication attempt. Although psychologists have found extensive evidence that contextual factors alter behavior, some have argued that context is unlikely to influence the results of direct replications precisely because these studies use the same methods as those used in the original research. To help resolve this debate, we recoded the 100 original studies from the Reproducibility Project on the extent to which the research topic of each study was contextually sensitive. Results suggested that the contextual sensitivity of the research topic was associated with replication success, even after statistically adjusting for several methodological characteristics (e.g., statistical power, effect size). The association between contextual sensitivity and replication success did not differ across psychological subdisciplines. These results suggest that researchers, replicators, and consumers should bemindful of contextual factors that might influence a psychological process. We offer several guidelines for dealing with contextual sensitivity in reproducibility.

AB - In recent years, scientists have paid increasing attention to reproducibility. For example, the Reproducibility Project, a large-scale replication attempt of 100 studies published in top psychology journals found that only 39% could be unambiguously reproduced. There is a growing consensus among scientists that the lack of reproducibility in psychology and other fields stems from various methodological factors, including low statistical power, researcher's degrees of freedom, and an emphasis on publishing surprising positive results. However, there is a contentious debate about the extent to which failures to reproduce certain results might also reflect contextual differences (often termed "hiddenmoderators") between the original research and the replication attempt. Although psychologists have found extensive evidence that contextual factors alter behavior, some have argued that context is unlikely to influence the results of direct replications precisely because these studies use the same methods as those used in the original research. To help resolve this debate, we recoded the 100 original studies from the Reproducibility Project on the extent to which the research topic of each study was contextually sensitive. Results suggested that the contextual sensitivity of the research topic was associated with replication success, even after statistically adjusting for several methodological characteristics (e.g., statistical power, effect size). The association between contextual sensitivity and replication success did not differ across psychological subdisciplines. These results suggest that researchers, replicators, and consumers should bemindful of contextual factors that might influence a psychological process. We offer several guidelines for dealing with contextual sensitivity in reproducibility.

KW - Context

KW - Meta-science

KW - Psychology

KW - Replication

KW - Reproducibility

UR - http://www.scopus.com/inward/record.url?scp=84973316189&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84973316189&partnerID=8YFLogxK

U2 - 10.1073/pnas.1521897113

DO - 10.1073/pnas.1521897113

M3 - Article

VL - 113

SP - 6454

EP - 6459

JO - Proceedings of the National Academy of Sciences of the United States of America

JF - Proceedings of the National Academy of Sciences of the United States of America

SN - 0027-8424

IS - 23

ER -