How Accurate Are Survey Responses on Social Media and Politics?

Andrew Guess, Kevin Munger, Jonathan Nagler, Joshua Tucker

Research output: Contribution to journalArticle

Abstract

How accurate are survey-based measures of social media use, in particular about political topics? We answer this question by linking original survey data collected during the U.S. 2016 election campaign with respondents’ observed social media activity. We use supervised machine learning to classify whether these Twitter and Facebook account data are content related to politics. We then benchmark our survey measures on frequency of posting about politics and the number of political figures followed. We find that, on average, our self-reported survey measures tend to correlate with observed social media activity. At the same time, we also find a worrying amount of individual-level discrepancy and problems related to extreme outliers. Our recommendations are twofold. The first is for survey questions about social media use to provide respondents with options covering a wider range of activity, especially in the long tail. The second is for survey questions to include specific content and anchors defining what it means for a post to be “about politics.”.

Original languageEnglish (US)
JournalPolitical Communication
DOIs
StateAccepted/In press - Jan 1 2018

Fingerprint

social media
politics
election campaign
twitter
facebook
Anchors
Learning systems
learning

Keywords

  • media effects
  • social media
  • survey measurement

ASJC Scopus subject areas

  • Communication
  • Sociology and Political Science

Cite this

How Accurate Are Survey Responses on Social Media and Politics? / Guess, Andrew; Munger, Kevin; Nagler, Jonathan; Tucker, Joshua.

In: Political Communication, 01.01.2018.

Research output: Contribution to journalArticle

@article{4c1cb92c7f8540e5bfbfe7d9311de7d6,
title = "How Accurate Are Survey Responses on Social Media and Politics?",
abstract = "How accurate are survey-based measures of social media use, in particular about political topics? We answer this question by linking original survey data collected during the U.S. 2016 election campaign with respondents’ observed social media activity. We use supervised machine learning to classify whether these Twitter and Facebook account data are content related to politics. We then benchmark our survey measures on frequency of posting about politics and the number of political figures followed. We find that, on average, our self-reported survey measures tend to correlate with observed social media activity. At the same time, we also find a worrying amount of individual-level discrepancy and problems related to extreme outliers. Our recommendations are twofold. The first is for survey questions about social media use to provide respondents with options covering a wider range of activity, especially in the long tail. The second is for survey questions to include specific content and anchors defining what it means for a post to be “about politics.”.",
keywords = "media effects, social media, survey measurement",
author = "Andrew Guess and Kevin Munger and Jonathan Nagler and Joshua Tucker",
year = "2018",
month = "1",
day = "1",
doi = "10.1080/10584609.2018.1504840",
language = "English (US)",
journal = "Political Communication",
issn = "1058-4609",
publisher = "Taylor and Francis Ltd.",

}

TY - JOUR

T1 - How Accurate Are Survey Responses on Social Media and Politics?

AU - Guess, Andrew

AU - Munger, Kevin

AU - Nagler, Jonathan

AU - Tucker, Joshua

PY - 2018/1/1

Y1 - 2018/1/1

N2 - How accurate are survey-based measures of social media use, in particular about political topics? We answer this question by linking original survey data collected during the U.S. 2016 election campaign with respondents’ observed social media activity. We use supervised machine learning to classify whether these Twitter and Facebook account data are content related to politics. We then benchmark our survey measures on frequency of posting about politics and the number of political figures followed. We find that, on average, our self-reported survey measures tend to correlate with observed social media activity. At the same time, we also find a worrying amount of individual-level discrepancy and problems related to extreme outliers. Our recommendations are twofold. The first is for survey questions about social media use to provide respondents with options covering a wider range of activity, especially in the long tail. The second is for survey questions to include specific content and anchors defining what it means for a post to be “about politics.”.

AB - How accurate are survey-based measures of social media use, in particular about political topics? We answer this question by linking original survey data collected during the U.S. 2016 election campaign with respondents’ observed social media activity. We use supervised machine learning to classify whether these Twitter and Facebook account data are content related to politics. We then benchmark our survey measures on frequency of posting about politics and the number of political figures followed. We find that, on average, our self-reported survey measures tend to correlate with observed social media activity. At the same time, we also find a worrying amount of individual-level discrepancy and problems related to extreme outliers. Our recommendations are twofold. The first is for survey questions about social media use to provide respondents with options covering a wider range of activity, especially in the long tail. The second is for survey questions to include specific content and anchors defining what it means for a post to be “about politics.”.

KW - media effects

KW - social media

KW - survey measurement

UR - http://www.scopus.com/inward/record.url?scp=85056107175&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85056107175&partnerID=8YFLogxK

U2 - 10.1080/10584609.2018.1504840

DO - 10.1080/10584609.2018.1504840

M3 - Article

AN - SCOPUS:85056107175

JO - Political Communication

JF - Political Communication

SN - 1058-4609

ER -