Saliency-based sequential image attention with multiset prediction

Sean Welleck, Jialin Mao, Kyunghyun Cho, Zheng Zhang

Research output: Contribution to journalConference article

Abstract

Humans process visual scenes selectively and sequentially using attention. Central to models of human visual attention is the saliency map. We propose a hierarchical visual architecture that operates on a saliency map and uses a novel attention mechanism to sequentially focus on salient regions and take additional glimpses within those regions. The architecture is motivated by human visual attention, and is used for multi-label image classification on a novel multiset task, demonstrating that it achieves high precision and recall while localizing objects with its attention. Unlike conventional multi-label image classification models, the model supports multiset prediction due to a reinforcement-learning based training process that allows for arbitrary label permutation and multiple instances per label.

Original languageEnglish (US)
Pages (from-to)5174-5184
Number of pages11
JournalAdvances in Neural Information Processing Systems
Volume2017-December
StatePublished - Jan 1 2017
Event31st Annual Conference on Neural Information Processing Systems, NIPS 2017 - Long Beach, United States
Duration: Dec 4 2017Dec 9 2017

Fingerprint

Labels
Image classification
Reinforcement learning

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Information Systems
  • Signal Processing

Cite this

Saliency-based sequential image attention with multiset prediction. / Welleck, Sean; Mao, Jialin; Cho, Kyunghyun; Zhang, Zheng.

In: Advances in Neural Information Processing Systems, Vol. 2017-December, 01.01.2017, p. 5174-5184.

Research output: Contribution to journalConference article

Welleck, Sean ; Mao, Jialin ; Cho, Kyunghyun ; Zhang, Zheng. / Saliency-based sequential image attention with multiset prediction. In: Advances in Neural Information Processing Systems. 2017 ; Vol. 2017-December. pp. 5174-5184.
@article{ff8c3851118b44108de64eb90fa85830,
title = "Saliency-based sequential image attention with multiset prediction",
abstract = "Humans process visual scenes selectively and sequentially using attention. Central to models of human visual attention is the saliency map. We propose a hierarchical visual architecture that operates on a saliency map and uses a novel attention mechanism to sequentially focus on salient regions and take additional glimpses within those regions. The architecture is motivated by human visual attention, and is used for multi-label image classification on a novel multiset task, demonstrating that it achieves high precision and recall while localizing objects with its attention. Unlike conventional multi-label image classification models, the model supports multiset prediction due to a reinforcement-learning based training process that allows for arbitrary label permutation and multiple instances per label.",
author = "Sean Welleck and Jialin Mao and Kyunghyun Cho and Zheng Zhang",
year = "2017",
month = "1",
day = "1",
language = "English (US)",
volume = "2017-December",
pages = "5174--5184",
journal = "Advances in Neural Information Processing Systems",
issn = "1049-5258",

}

TY - JOUR

T1 - Saliency-based sequential image attention with multiset prediction

AU - Welleck, Sean

AU - Mao, Jialin

AU - Cho, Kyunghyun

AU - Zhang, Zheng

PY - 2017/1/1

Y1 - 2017/1/1

N2 - Humans process visual scenes selectively and sequentially using attention. Central to models of human visual attention is the saliency map. We propose a hierarchical visual architecture that operates on a saliency map and uses a novel attention mechanism to sequentially focus on salient regions and take additional glimpses within those regions. The architecture is motivated by human visual attention, and is used for multi-label image classification on a novel multiset task, demonstrating that it achieves high precision and recall while localizing objects with its attention. Unlike conventional multi-label image classification models, the model supports multiset prediction due to a reinforcement-learning based training process that allows for arbitrary label permutation and multiple instances per label.

AB - Humans process visual scenes selectively and sequentially using attention. Central to models of human visual attention is the saliency map. We propose a hierarchical visual architecture that operates on a saliency map and uses a novel attention mechanism to sequentially focus on salient regions and take additional glimpses within those regions. The architecture is motivated by human visual attention, and is used for multi-label image classification on a novel multiset task, demonstrating that it achieves high precision and recall while localizing objects with its attention. Unlike conventional multi-label image classification models, the model supports multiset prediction due to a reinforcement-learning based training process that allows for arbitrary label permutation and multiple instances per label.

UR - http://www.scopus.com/inward/record.url?scp=85047020556&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85047020556&partnerID=8YFLogxK

M3 - Conference article

AN - SCOPUS:85047020556

VL - 2017-December

SP - 5174

EP - 5184

JO - Advances in Neural Information Processing Systems

JF - Advances in Neural Information Processing Systems

SN - 1049-5258

ER -