Leveraging known data for missing label prediction in cultural heritage context

Abdelhak Belhi, Abdelaziz Bouras, Sebti Foufou

Research output: Contribution to journalArticle

Abstract

Cultural heritage represents a reliable medium for history and knowledge transfer. Cultural heritage assets are often exhibited in museums and heritage sites all over the world. However, many assets are poorly labeled, which decreases their historical value. If an asset's history is lost, its historical value is also lost. The classification and annotation of overlooked or incomplete cultural assets increase their historical value and allows the discovery of various types of historical links. In this paper, we tackle the challenge of automatically classifying and annotating cultural heritage assets using their visual features as well as the metadata available at hand. Traditional approaches mainly rely only on image data and machine-learning-based techniques to predict missing labels. Often, visual data are not the only information available at hand. In this paper, we present a novel multimodal classification approach for cultural heritage assets that relies on a multitask neural network where a convolutional neural network (CNN) is designed for visual feature learning and a regular neural network is used for textual feature learning. These networks are merged and trained using a shared loss. The combined networks rely on both image and textual features to achieve better asset classification. Initial tests related to painting assets showed that our approach performs better than traditional CNNs that only rely on images as input.

Original languageEnglish (US)
Article number1768
JournalApplied Sciences (Switzerland)
Volume8
Issue number10
DOIs
StatePublished - Sep 30 2018

Fingerprint

Labels
Neural networks
predictions
History
Museums
learning
Painting
Metadata
Learning systems
histories
annotations
metadata
museums
machine learning
classifying

Keywords

  • Convolutional neural networks
  • Cultural heritage
  • Digital heritage
  • Digital preservation
  • Multimodal classification

ASJC Scopus subject areas

  • Materials Science(all)
  • Instrumentation
  • Engineering(all)
  • Process Chemistry and Technology
  • Computer Science Applications
  • Fluid Flow and Transfer Processes

Cite this

Leveraging known data for missing label prediction in cultural heritage context. / Belhi, Abdelhak; Bouras, Abdelaziz; Foufou, Sebti.

In: Applied Sciences (Switzerland), Vol. 8, No. 10, 1768, 30.09.2018.

Research output: Contribution to journalArticle

@article{c8e2fb0ca5bb489fa1a88b7c38ce569e,
title = "Leveraging known data for missing label prediction in cultural heritage context",
abstract = "Cultural heritage represents a reliable medium for history and knowledge transfer. Cultural heritage assets are often exhibited in museums and heritage sites all over the world. However, many assets are poorly labeled, which decreases their historical value. If an asset's history is lost, its historical value is also lost. The classification and annotation of overlooked or incomplete cultural assets increase their historical value and allows the discovery of various types of historical links. In this paper, we tackle the challenge of automatically classifying and annotating cultural heritage assets using their visual features as well as the metadata available at hand. Traditional approaches mainly rely only on image data and machine-learning-based techniques to predict missing labels. Often, visual data are not the only information available at hand. In this paper, we present a novel multimodal classification approach for cultural heritage assets that relies on a multitask neural network where a convolutional neural network (CNN) is designed for visual feature learning and a regular neural network is used for textual feature learning. These networks are merged and trained using a shared loss. The combined networks rely on both image and textual features to achieve better asset classification. Initial tests related to painting assets showed that our approach performs better than traditional CNNs that only rely on images as input.",
keywords = "Convolutional neural networks, Cultural heritage, Digital heritage, Digital preservation, Multimodal classification",
author = "Abdelhak Belhi and Abdelaziz Bouras and Sebti Foufou",
year = "2018",
month = "9",
day = "30",
doi = "10.3390/app8101768",
language = "English (US)",
volume = "8",
journal = "Applied Sciences (Switzerland)",
issn = "2076-3417",
publisher = "Multidisciplinary Digital Publishing Institute (MDPI)",
number = "10",

}

TY - JOUR

T1 - Leveraging known data for missing label prediction in cultural heritage context

AU - Belhi, Abdelhak

AU - Bouras, Abdelaziz

AU - Foufou, Sebti

PY - 2018/9/30

Y1 - 2018/9/30

N2 - Cultural heritage represents a reliable medium for history and knowledge transfer. Cultural heritage assets are often exhibited in museums and heritage sites all over the world. However, many assets are poorly labeled, which decreases their historical value. If an asset's history is lost, its historical value is also lost. The classification and annotation of overlooked or incomplete cultural assets increase their historical value and allows the discovery of various types of historical links. In this paper, we tackle the challenge of automatically classifying and annotating cultural heritage assets using their visual features as well as the metadata available at hand. Traditional approaches mainly rely only on image data and machine-learning-based techniques to predict missing labels. Often, visual data are not the only information available at hand. In this paper, we present a novel multimodal classification approach for cultural heritage assets that relies on a multitask neural network where a convolutional neural network (CNN) is designed for visual feature learning and a regular neural network is used for textual feature learning. These networks are merged and trained using a shared loss. The combined networks rely on both image and textual features to achieve better asset classification. Initial tests related to painting assets showed that our approach performs better than traditional CNNs that only rely on images as input.

AB - Cultural heritage represents a reliable medium for history and knowledge transfer. Cultural heritage assets are often exhibited in museums and heritage sites all over the world. However, many assets are poorly labeled, which decreases their historical value. If an asset's history is lost, its historical value is also lost. The classification and annotation of overlooked or incomplete cultural assets increase their historical value and allows the discovery of various types of historical links. In this paper, we tackle the challenge of automatically classifying and annotating cultural heritage assets using their visual features as well as the metadata available at hand. Traditional approaches mainly rely only on image data and machine-learning-based techniques to predict missing labels. Often, visual data are not the only information available at hand. In this paper, we present a novel multimodal classification approach for cultural heritage assets that relies on a multitask neural network where a convolutional neural network (CNN) is designed for visual feature learning and a regular neural network is used for textual feature learning. These networks are merged and trained using a shared loss. The combined networks rely on both image and textual features to achieve better asset classification. Initial tests related to painting assets showed that our approach performs better than traditional CNNs that only rely on images as input.

KW - Convolutional neural networks

KW - Cultural heritage

KW - Digital heritage

KW - Digital preservation

KW - Multimodal classification

UR - http://www.scopus.com/inward/record.url?scp=85054092800&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85054092800&partnerID=8YFLogxK

U2 - 10.3390/app8101768

DO - 10.3390/app8101768

M3 - Article

AN - SCOPUS:85054092800

VL - 8

JO - Applied Sciences (Switzerland)

JF - Applied Sciences (Switzerland)

SN - 2076-3417

IS - 10

M1 - 1768

ER -