User conditional hashtag prediction for images

Emily Denton, Jason Weston, Manohar Paluri, Lubomir Bourdev, Robert Fergus

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Understanding the content of user's image posts is a particularly interesting problem in social networks and web settings. Current machine learning techniques focus mostly on curated training sets of image-label pairs, and perform image classification given the pixels within the image. In this work we instead leverage the wealth of information available from users: firstly, we employ user hashtags to capture the description of image content; and secondly, we make use of valuable contextual information about the user. We show how user metadata (age, gender, etc.) combined with image features derived from a convolutional neural network can be used to perform hashtag prediction. We explore two ways of combining these heterogeneous features into a learning framework: (i) simple concatenation; and (ii) a 3-way multiplicative gating, where the image model is conditioned on the user metadata. We apply these models to a large dataset of de-identified Facebook posts and demonstrate that modeling the user can significantly improve the tag prediction quality over current state-of-the-art methods.

Original languageEnglish (US)
Title of host publicationKDD 2015 - Proceedings of the 21st ACM SIGKDD Conference on Knowledge Discovery and Data Mining
PublisherAssociation for Computing Machinery
Pages1731-1740
Number of pages10
Volume2015-August
ISBN (Print)9781450336642
DOIs
StatePublished - Aug 10 2015
Event21st ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2015 - Sydney, Australia
Duration: Aug 10 2015Aug 13 2015

Other

Other21st ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2015
CountryAustralia
CitySydney
Period8/10/158/13/15

Fingerprint

Metadata
Image classification
Learning systems
Labels
Pixels
Neural networks

Keywords

  • Deep learning
  • Hashtagging
  • Large scale image annotation
  • Social media
  • User modeling

ASJC Scopus subject areas

  • Software
  • Information Systems

Cite this

Denton, E., Weston, J., Paluri, M., Bourdev, L., & Fergus, R. (2015). User conditional hashtag prediction for images. In KDD 2015 - Proceedings of the 21st ACM SIGKDD Conference on Knowledge Discovery and Data Mining (Vol. 2015-August, pp. 1731-1740). Association for Computing Machinery. https://doi.org/10.1145/2783258.2788576

User conditional hashtag prediction for images. / Denton, Emily; Weston, Jason; Paluri, Manohar; Bourdev, Lubomir; Fergus, Robert.

KDD 2015 - Proceedings of the 21st ACM SIGKDD Conference on Knowledge Discovery and Data Mining. Vol. 2015-August Association for Computing Machinery, 2015. p. 1731-1740.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Denton, E, Weston, J, Paluri, M, Bourdev, L & Fergus, R 2015, User conditional hashtag prediction for images. in KDD 2015 - Proceedings of the 21st ACM SIGKDD Conference on Knowledge Discovery and Data Mining. vol. 2015-August, Association for Computing Machinery, pp. 1731-1740, 21st ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2015, Sydney, Australia, 8/10/15. https://doi.org/10.1145/2783258.2788576
Denton E, Weston J, Paluri M, Bourdev L, Fergus R. User conditional hashtag prediction for images. In KDD 2015 - Proceedings of the 21st ACM SIGKDD Conference on Knowledge Discovery and Data Mining. Vol. 2015-August. Association for Computing Machinery. 2015. p. 1731-1740 https://doi.org/10.1145/2783258.2788576
Denton, Emily ; Weston, Jason ; Paluri, Manohar ; Bourdev, Lubomir ; Fergus, Robert. / User conditional hashtag prediction for images. KDD 2015 - Proceedings of the 21st ACM SIGKDD Conference on Knowledge Discovery and Data Mining. Vol. 2015-August Association for Computing Machinery, 2015. pp. 1731-1740
@inproceedings{8fdac0243af449b4b30eba72acb9c12a,
title = "User conditional hashtag prediction for images",
abstract = "Understanding the content of user's image posts is a particularly interesting problem in social networks and web settings. Current machine learning techniques focus mostly on curated training sets of image-label pairs, and perform image classification given the pixels within the image. In this work we instead leverage the wealth of information available from users: firstly, we employ user hashtags to capture the description of image content; and secondly, we make use of valuable contextual information about the user. We show how user metadata (age, gender, etc.) combined with image features derived from a convolutional neural network can be used to perform hashtag prediction. We explore two ways of combining these heterogeneous features into a learning framework: (i) simple concatenation; and (ii) a 3-way multiplicative gating, where the image model is conditioned on the user metadata. We apply these models to a large dataset of de-identified Facebook posts and demonstrate that modeling the user can significantly improve the tag prediction quality over current state-of-the-art methods.",
keywords = "Deep learning, Hashtagging, Large scale image annotation, Social media, User modeling",
author = "Emily Denton and Jason Weston and Manohar Paluri and Lubomir Bourdev and Robert Fergus",
year = "2015",
month = "8",
day = "10",
doi = "10.1145/2783258.2788576",
language = "English (US)",
isbn = "9781450336642",
volume = "2015-August",
pages = "1731--1740",
booktitle = "KDD 2015 - Proceedings of the 21st ACM SIGKDD Conference on Knowledge Discovery and Data Mining",
publisher = "Association for Computing Machinery",

}

TY - GEN

T1 - User conditional hashtag prediction for images

AU - Denton, Emily

AU - Weston, Jason

AU - Paluri, Manohar

AU - Bourdev, Lubomir

AU - Fergus, Robert

PY - 2015/8/10

Y1 - 2015/8/10

N2 - Understanding the content of user's image posts is a particularly interesting problem in social networks and web settings. Current machine learning techniques focus mostly on curated training sets of image-label pairs, and perform image classification given the pixels within the image. In this work we instead leverage the wealth of information available from users: firstly, we employ user hashtags to capture the description of image content; and secondly, we make use of valuable contextual information about the user. We show how user metadata (age, gender, etc.) combined with image features derived from a convolutional neural network can be used to perform hashtag prediction. We explore two ways of combining these heterogeneous features into a learning framework: (i) simple concatenation; and (ii) a 3-way multiplicative gating, where the image model is conditioned on the user metadata. We apply these models to a large dataset of de-identified Facebook posts and demonstrate that modeling the user can significantly improve the tag prediction quality over current state-of-the-art methods.

AB - Understanding the content of user's image posts is a particularly interesting problem in social networks and web settings. Current machine learning techniques focus mostly on curated training sets of image-label pairs, and perform image classification given the pixels within the image. In this work we instead leverage the wealth of information available from users: firstly, we employ user hashtags to capture the description of image content; and secondly, we make use of valuable contextual information about the user. We show how user metadata (age, gender, etc.) combined with image features derived from a convolutional neural network can be used to perform hashtag prediction. We explore two ways of combining these heterogeneous features into a learning framework: (i) simple concatenation; and (ii) a 3-way multiplicative gating, where the image model is conditioned on the user metadata. We apply these models to a large dataset of de-identified Facebook posts and demonstrate that modeling the user can significantly improve the tag prediction quality over current state-of-the-art methods.

KW - Deep learning

KW - Hashtagging

KW - Large scale image annotation

KW - Social media

KW - User modeling

UR - http://www.scopus.com/inward/record.url?scp=84954167407&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84954167407&partnerID=8YFLogxK

U2 - 10.1145/2783258.2788576

DO - 10.1145/2783258.2788576

M3 - Conference contribution

AN - SCOPUS:84954167407

SN - 9781450336642

VL - 2015-August

SP - 1731

EP - 1740

BT - KDD 2015 - Proceedings of the 21st ACM SIGKDD Conference on Knowledge Discovery and Data Mining

PB - Association for Computing Machinery

ER -