Inaccurate estimation of disparities due to mischievous responders: Several suggestions to assess conclusions

Research output: Contribution to journalArticle

Abstract

This article introduces novel sensitivity-analysis procedures for investigating and reducing the bias that mischievous responders (i.e., youths who provide extreme, and potentially untruthful, responses to multiple questions) often introduce in adolescent disparity estimates based on data from self-administered questionnaires (SAQs). Mischievous responders affect a wide range of disparity estimates, including those between adoptees and nonadoptees, sexual minorities and nonminorities, and individuals with and without disabilities. Thus, the procedures introduced here have broad relevance to research and can be widely, and easily, implemented. The sensitivity-analysis procedures are illustrated with SAQ data from youths in Grades 9-12 (N =11,829) to examine between-group disparities based on sexual identity, gender identity, and physical disability. Sensitivity analyses revealed that each disparity estimated with these data was extremely sensitive to the presence of potentially mischievous responders. Patterns were similar across multiple approaches to dealing with mischievous responders, across various outcomes, and across different between-group comparisons. Mischievous responders are ubiquitous in adolescent research using SAQs and can, even in small numbers, lead to inaccurate conclusions that substantively affect research, policy, and public discourse regarding a variety of disparities. This article calls attention to this widespread problem and provides practical suggestions for assessing it, even when data are already collected.

Original languageEnglish (US)
Pages (from-to)171-185
Number of pages15
JournalEducational Researcher
Volume43
Issue number4
DOIs
StatePublished - 2014

Fingerprint

analysis procedure
questionnaire
adolescent
physical disability
research policy
Group
school grade
disability
minority
discourse
gender
trend

Keywords

  • Adolescents
  • And questioning
  • Bisexual
  • Disparities
  • Equity
  • Evaluation
  • Gay
  • Lesbian
  • Mischievous responders
  • Physical disabilities
  • Questionnaires
  • Self report
  • Sensitivity analysis
  • Survey research
  • Transgender

ASJC Scopus subject areas

  • Education

Cite this

@article{202629ec5edf4265a9a3609f3eaea039,
title = "Inaccurate estimation of disparities due to mischievous responders: Several suggestions to assess conclusions",
abstract = "This article introduces novel sensitivity-analysis procedures for investigating and reducing the bias that mischievous responders (i.e., youths who provide extreme, and potentially untruthful, responses to multiple questions) often introduce in adolescent disparity estimates based on data from self-administered questionnaires (SAQs). Mischievous responders affect a wide range of disparity estimates, including those between adoptees and nonadoptees, sexual minorities and nonminorities, and individuals with and without disabilities. Thus, the procedures introduced here have broad relevance to research and can be widely, and easily, implemented. The sensitivity-analysis procedures are illustrated with SAQ data from youths in Grades 9-12 (N =11,829) to examine between-group disparities based on sexual identity, gender identity, and physical disability. Sensitivity analyses revealed that each disparity estimated with these data was extremely sensitive to the presence of potentially mischievous responders. Patterns were similar across multiple approaches to dealing with mischievous responders, across various outcomes, and across different between-group comparisons. Mischievous responders are ubiquitous in adolescent research using SAQs and can, even in small numbers, lead to inaccurate conclusions that substantively affect research, policy, and public discourse regarding a variety of disparities. This article calls attention to this widespread problem and provides practical suggestions for assessing it, even when data are already collected.",
keywords = "Adolescents, And questioning, Bisexual, Disparities, Equity, Evaluation, Gay, Lesbian, Mischievous responders, Physical disabilities, Questionnaires, Self report, Sensitivity analysis, Survey research, Transgender",
author = "Robinson-Cimpian, {Joseph P.}",
year = "2014",
doi = "10.3102/0013189X14534297",
language = "English (US)",
volume = "43",
pages = "171--185",
journal = "Educational Researcher",
issn = "0013-189X",
publisher = "SAGE Publications Inc.",
number = "4",

}

TY - JOUR

T1 - Inaccurate estimation of disparities due to mischievous responders

T2 - Several suggestions to assess conclusions

AU - Robinson-Cimpian, Joseph P.

PY - 2014

Y1 - 2014

N2 - This article introduces novel sensitivity-analysis procedures for investigating and reducing the bias that mischievous responders (i.e., youths who provide extreme, and potentially untruthful, responses to multiple questions) often introduce in adolescent disparity estimates based on data from self-administered questionnaires (SAQs). Mischievous responders affect a wide range of disparity estimates, including those between adoptees and nonadoptees, sexual minorities and nonminorities, and individuals with and without disabilities. Thus, the procedures introduced here have broad relevance to research and can be widely, and easily, implemented. The sensitivity-analysis procedures are illustrated with SAQ data from youths in Grades 9-12 (N =11,829) to examine between-group disparities based on sexual identity, gender identity, and physical disability. Sensitivity analyses revealed that each disparity estimated with these data was extremely sensitive to the presence of potentially mischievous responders. Patterns were similar across multiple approaches to dealing with mischievous responders, across various outcomes, and across different between-group comparisons. Mischievous responders are ubiquitous in adolescent research using SAQs and can, even in small numbers, lead to inaccurate conclusions that substantively affect research, policy, and public discourse regarding a variety of disparities. This article calls attention to this widespread problem and provides practical suggestions for assessing it, even when data are already collected.

AB - This article introduces novel sensitivity-analysis procedures for investigating and reducing the bias that mischievous responders (i.e., youths who provide extreme, and potentially untruthful, responses to multiple questions) often introduce in adolescent disparity estimates based on data from self-administered questionnaires (SAQs). Mischievous responders affect a wide range of disparity estimates, including those between adoptees and nonadoptees, sexual minorities and nonminorities, and individuals with and without disabilities. Thus, the procedures introduced here have broad relevance to research and can be widely, and easily, implemented. The sensitivity-analysis procedures are illustrated with SAQ data from youths in Grades 9-12 (N =11,829) to examine between-group disparities based on sexual identity, gender identity, and physical disability. Sensitivity analyses revealed that each disparity estimated with these data was extremely sensitive to the presence of potentially mischievous responders. Patterns were similar across multiple approaches to dealing with mischievous responders, across various outcomes, and across different between-group comparisons. Mischievous responders are ubiquitous in adolescent research using SAQs and can, even in small numbers, lead to inaccurate conclusions that substantively affect research, policy, and public discourse regarding a variety of disparities. This article calls attention to this widespread problem and provides practical suggestions for assessing it, even when data are already collected.

KW - Adolescents

KW - And questioning

KW - Bisexual

KW - Disparities

KW - Equity

KW - Evaluation

KW - Gay

KW - Lesbian

KW - Mischievous responders

KW - Physical disabilities

KW - Questionnaires

KW - Self report

KW - Sensitivity analysis

KW - Survey research

KW - Transgender

UR - http://www.scopus.com/inward/record.url?scp=84905044597&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84905044597&partnerID=8YFLogxK

U2 - 10.3102/0013189X14534297

DO - 10.3102/0013189X14534297

M3 - Article

AN - SCOPUS:84905044597

VL - 43

SP - 171

EP - 185

JO - Educational Researcher

JF - Educational Researcher

SN - 0013-189X

IS - 4

ER -