Leveraging known data for missing label prediction in cultural heritage context

Abdelhak Belhi, Abdelaziz Bouras, Sebti Foufou

    Research output: Contribution to journalArticle

    Abstract

    Cultural heritage represents a reliable medium for history and knowledge transfer. Cultural heritage assets are often exhibited in museums and heritage sites all over the world. However, many assets are poorly labeled, which decreases their historical value. If an asset's history is lost, its historical value is also lost. The classification and annotation of overlooked or incomplete cultural assets increase their historical value and allows the discovery of various types of historical links. In this paper, we tackle the challenge of automatically classifying and annotating cultural heritage assets using their visual features as well as the metadata available at hand. Traditional approaches mainly rely only on image data and machine-learning-based techniques to predict missing labels. Often, visual data are not the only information available at hand. In this paper, we present a novel multimodal classification approach for cultural heritage assets that relies on a multitask neural network where a convolutional neural network (CNN) is designed for visual feature learning and a regular neural network is used for textual feature learning. These networks are merged and trained using a shared loss. The combined networks rely on both image and textual features to achieve better asset classification. Initial tests related to painting assets showed that our approach performs better than traditional CNNs that only rely on images as input.

    Original languageEnglish (US)
    Article number1768
    JournalApplied Sciences (Switzerland)
    Volume8
    Issue number10
    DOIs
    StatePublished - Sep 30 2018

    Fingerprint

    Labels
    Neural networks
    predictions
    History
    Museums
    learning
    Painting
    Metadata
    Learning systems
    histories
    annotations
    metadata
    museums
    machine learning
    classifying

    Keywords

    • Convolutional neural networks
    • Cultural heritage
    • Digital heritage
    • Digital preservation
    • Multimodal classification

    ASJC Scopus subject areas

    • Materials Science(all)
    • Instrumentation
    • Engineering(all)
    • Process Chemistry and Technology
    • Computer Science Applications
    • Fluid Flow and Transfer Processes

    Cite this

    Leveraging known data for missing label prediction in cultural heritage context. / Belhi, Abdelhak; Bouras, Abdelaziz; Foufou, Sebti.

    In: Applied Sciences (Switzerland), Vol. 8, No. 10, 1768, 30.09.2018.

    Research output: Contribution to journalArticle

    Belhi, Abdelhak ; Bouras, Abdelaziz ; Foufou, Sebti. / Leveraging known data for missing label prediction in cultural heritage context. In: Applied Sciences (Switzerland). 2018 ; Vol. 8, No. 10.
    @article{d3178826c8944923871cbe0f4563ad74,
    title = "Leveraging known data for missing label prediction in cultural heritage context",
    abstract = "Cultural heritage represents a reliable medium for history and knowledge transfer. Cultural heritage assets are often exhibited in museums and heritage sites all over the world. However, many assets are poorly labeled, which decreases their historical value. If an asset's history is lost, its historical value is also lost. The classification and annotation of overlooked or incomplete cultural assets increase their historical value and allows the discovery of various types of historical links. In this paper, we tackle the challenge of automatically classifying and annotating cultural heritage assets using their visual features as well as the metadata available at hand. Traditional approaches mainly rely only on image data and machine-learning-based techniques to predict missing labels. Often, visual data are not the only information available at hand. In this paper, we present a novel multimodal classification approach for cultural heritage assets that relies on a multitask neural network where a convolutional neural network (CNN) is designed for visual feature learning and a regular neural network is used for textual feature learning. These networks are merged and trained using a shared loss. The combined networks rely on both image and textual features to achieve better asset classification. Initial tests related to painting assets showed that our approach performs better than traditional CNNs that only rely on images as input.",
    keywords = "Convolutional neural networks, Cultural heritage, Digital heritage, Digital preservation, Multimodal classification",
    author = "Abdelhak Belhi and Abdelaziz Bouras and Sebti Foufou",
    year = "2018",
    month = "9",
    day = "30",
    doi = "10.3390/app8101768",
    language = "English (US)",
    volume = "8",
    journal = "Applied Sciences (Switzerland)",
    issn = "2076-3417",
    publisher = "Multidisciplinary Digital Publishing Institute (MDPI)",
    number = "10",

    }

    TY - JOUR

    T1 - Leveraging known data for missing label prediction in cultural heritage context

    AU - Belhi, Abdelhak

    AU - Bouras, Abdelaziz

    AU - Foufou, Sebti

    PY - 2018/9/30

    Y1 - 2018/9/30

    N2 - Cultural heritage represents a reliable medium for history and knowledge transfer. Cultural heritage assets are often exhibited in museums and heritage sites all over the world. However, many assets are poorly labeled, which decreases their historical value. If an asset's history is lost, its historical value is also lost. The classification and annotation of overlooked or incomplete cultural assets increase their historical value and allows the discovery of various types of historical links. In this paper, we tackle the challenge of automatically classifying and annotating cultural heritage assets using their visual features as well as the metadata available at hand. Traditional approaches mainly rely only on image data and machine-learning-based techniques to predict missing labels. Often, visual data are not the only information available at hand. In this paper, we present a novel multimodal classification approach for cultural heritage assets that relies on a multitask neural network where a convolutional neural network (CNN) is designed for visual feature learning and a regular neural network is used for textual feature learning. These networks are merged and trained using a shared loss. The combined networks rely on both image and textual features to achieve better asset classification. Initial tests related to painting assets showed that our approach performs better than traditional CNNs that only rely on images as input.

    AB - Cultural heritage represents a reliable medium for history and knowledge transfer. Cultural heritage assets are often exhibited in museums and heritage sites all over the world. However, many assets are poorly labeled, which decreases their historical value. If an asset's history is lost, its historical value is also lost. The classification and annotation of overlooked or incomplete cultural assets increase their historical value and allows the discovery of various types of historical links. In this paper, we tackle the challenge of automatically classifying and annotating cultural heritage assets using their visual features as well as the metadata available at hand. Traditional approaches mainly rely only on image data and machine-learning-based techniques to predict missing labels. Often, visual data are not the only information available at hand. In this paper, we present a novel multimodal classification approach for cultural heritage assets that relies on a multitask neural network where a convolutional neural network (CNN) is designed for visual feature learning and a regular neural network is used for textual feature learning. These networks are merged and trained using a shared loss. The combined networks rely on both image and textual features to achieve better asset classification. Initial tests related to painting assets showed that our approach performs better than traditional CNNs that only rely on images as input.

    KW - Convolutional neural networks

    KW - Cultural heritage

    KW - Digital heritage

    KW - Digital preservation

    KW - Multimodal classification

    UR - http://www.scopus.com/inward/record.url?scp=85054092800&partnerID=8YFLogxK

    UR - http://www.scopus.com/inward/citedby.url?scp=85054092800&partnerID=8YFLogxK

    U2 - 10.3390/app8101768

    DO - 10.3390/app8101768

    M3 - Article

    VL - 8

    JO - Applied Sciences (Switzerland)

    JF - Applied Sciences (Switzerland)

    SN - 2076-3417

    IS - 10

    M1 - 1768

    ER -