Perceptual analysis of the f-measure for evaluating section boundaries in music: Proceedings of the 15th International Society for Music Information Retrieval Conference (ISMIR 2014)

Oriol Nieto, Morwaread Farbood, Tristan Jehan, Juan Bello

Research output: Contribution to conferencePaper

Abstract

In this paper, we aim to raise awareness of the limitations of the F-measure when evaluating the quality of the boundaries found in the automatic segmentation of music. We present and discuss the results of various experiments where subjects listened to different musical excerpts containing boundary indications and had to rate the quality of the boundaries. These boundaries were carefully generated from state-of-the-art segmentation algorithms as well as human-annotated data. The results show that humans tend to give more relevance to the precision component of the F-measure rather than the recall component, therefore making the classical F-measure not as perceptually informative as currently assumed. Based on the results of the experiments, we discuss the potential of an alternative evaluation based on the F-measure that emphasizes precision over recall, making the section boundary evaluation more expressive and reliable.

Original languageEnglish (US)
Pages265-270
Number of pages6
StatePublished - Jan 1 2014
Event15th International Society for Music Information Retrieval Conference, ISMIR 2014 - Taipei, Taiwan, Province of China
Duration: Oct 27 2014Oct 31 2014

Conference

Conference15th International Society for Music Information Retrieval Conference, ISMIR 2014
CountryTaiwan, Province of China
CityTaipei
Period10/27/1410/31/14

Fingerprint

Information retrieval
Experiments
Proceedings
Music
Music Information Retrieval
Evaluation
Experiment
Segmentation

ASJC Scopus subject areas

  • Music
  • Information Systems

Cite this

Nieto, O., Farbood, M., Jehan, T., & Bello, J. (2014). Perceptual analysis of the f-measure for evaluating section boundaries in music: Proceedings of the 15th International Society for Music Information Retrieval Conference (ISMIR 2014). 265-270. Paper presented at 15th International Society for Music Information Retrieval Conference, ISMIR 2014, Taipei, Taiwan, Province of China.

Perceptual analysis of the f-measure for evaluating section boundaries in music : Proceedings of the 15th International Society for Music Information Retrieval Conference (ISMIR 2014). / Nieto, Oriol; Farbood, Morwaread; Jehan, Tristan; Bello, Juan.

2014. 265-270 Paper presented at 15th International Society for Music Information Retrieval Conference, ISMIR 2014, Taipei, Taiwan, Province of China.

Research output: Contribution to conferencePaper

Nieto, O, Farbood, M, Jehan, T & Bello, J 2014, 'Perceptual analysis of the f-measure for evaluating section boundaries in music: Proceedings of the 15th International Society for Music Information Retrieval Conference (ISMIR 2014)', Paper presented at 15th International Society for Music Information Retrieval Conference, ISMIR 2014, Taipei, Taiwan, Province of China, 10/27/14 - 10/31/14 pp. 265-270.
Nieto O, Farbood M, Jehan T, Bello J. Perceptual analysis of the f-measure for evaluating section boundaries in music: Proceedings of the 15th International Society for Music Information Retrieval Conference (ISMIR 2014). 2014. Paper presented at 15th International Society for Music Information Retrieval Conference, ISMIR 2014, Taipei, Taiwan, Province of China.
Nieto, Oriol ; Farbood, Morwaread ; Jehan, Tristan ; Bello, Juan. / Perceptual analysis of the f-measure for evaluating section boundaries in music : Proceedings of the 15th International Society for Music Information Retrieval Conference (ISMIR 2014). Paper presented at 15th International Society for Music Information Retrieval Conference, ISMIR 2014, Taipei, Taiwan, Province of China.6 p.
@conference{85a9a2892b0a42f49f9824670ab961a8,
title = "Perceptual analysis of the f-measure for evaluating section boundaries in music: Proceedings of the 15th International Society for Music Information Retrieval Conference (ISMIR 2014)",
abstract = "In this paper, we aim to raise awareness of the limitations of the F-measure when evaluating the quality of the boundaries found in the automatic segmentation of music. We present and discuss the results of various experiments where subjects listened to different musical excerpts containing boundary indications and had to rate the quality of the boundaries. These boundaries were carefully generated from state-of-the-art segmentation algorithms as well as human-annotated data. The results show that humans tend to give more relevance to the precision component of the F-measure rather than the recall component, therefore making the classical F-measure not as perceptually informative as currently assumed. Based on the results of the experiments, we discuss the potential of an alternative evaluation based on the F-measure that emphasizes precision over recall, making the section boundary evaluation more expressive and reliable.",
author = "Oriol Nieto and Morwaread Farbood and Tristan Jehan and Juan Bello",
year = "2014",
month = "1",
day = "1",
language = "English (US)",
pages = "265--270",
note = "15th International Society for Music Information Retrieval Conference, ISMIR 2014 ; Conference date: 27-10-2014 Through 31-10-2014",

}

TY - CONF

T1 - Perceptual analysis of the f-measure for evaluating section boundaries in music

T2 - Proceedings of the 15th International Society for Music Information Retrieval Conference (ISMIR 2014)

AU - Nieto, Oriol

AU - Farbood, Morwaread

AU - Jehan, Tristan

AU - Bello, Juan

PY - 2014/1/1

Y1 - 2014/1/1

N2 - In this paper, we aim to raise awareness of the limitations of the F-measure when evaluating the quality of the boundaries found in the automatic segmentation of music. We present and discuss the results of various experiments where subjects listened to different musical excerpts containing boundary indications and had to rate the quality of the boundaries. These boundaries were carefully generated from state-of-the-art segmentation algorithms as well as human-annotated data. The results show that humans tend to give more relevance to the precision component of the F-measure rather than the recall component, therefore making the classical F-measure not as perceptually informative as currently assumed. Based on the results of the experiments, we discuss the potential of an alternative evaluation based on the F-measure that emphasizes precision over recall, making the section boundary evaluation more expressive and reliable.

AB - In this paper, we aim to raise awareness of the limitations of the F-measure when evaluating the quality of the boundaries found in the automatic segmentation of music. We present and discuss the results of various experiments where subjects listened to different musical excerpts containing boundary indications and had to rate the quality of the boundaries. These boundaries were carefully generated from state-of-the-art segmentation algorithms as well as human-annotated data. The results show that humans tend to give more relevance to the precision component of the F-measure rather than the recall component, therefore making the classical F-measure not as perceptually informative as currently assumed. Based on the results of the experiments, we discuss the potential of an alternative evaluation based on the F-measure that emphasizes precision over recall, making the section boundary evaluation more expressive and reliable.

UR - http://www.scopus.com/inward/record.url?scp=85066072277&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85066072277&partnerID=8YFLogxK

M3 - Paper

AN - SCOPUS:85066072277

SP - 265

EP - 270

ER -