Ranking feature sets for emotion models used in classroom based intelligent tutoring systems

David G. Cooper, Kasia Muldner, Ivon Arroyo, Beverly Park Woolf, Winslow Burleson

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Recent progress has been made by using sensors with Intelligent Tutoring Systems in classrooms in order to predict the affective state of students users. If tutors are able to interpret sensor data with new students based on past experience, rather than having to be individually trained, then this will enable tutor developers to evaluate various methods of adapting to each student's affective state using consistent predictions. In the past, our classifiers have predicted student emotions with an accuracy between 78% and 87%. However, it is still unclear which sensors are best, and the educational technology community needs to know this to develop better than baseline classifiers, e.g. ones that use only frequency of emotional occurrence to predict affective state. This paper suggests a method to clarify classifier ranking for the purpose of affective models. The method begins with a careful collection of a training and testing set, each from a separate population, and concludes with a non-parametric ranking of the trained classifiers on the testing set. We illustrate this method with classifiers trained on data collected in the Fall of 2008 and tested on data collected in the Spring of 2009. Our results show that the classifiers for some affective states are significantly better than the baseline model; a validation analysis showed that some but not all classifier rankings generalize to new settings. Overall, our analysis suggests that though there is some benefit gained from simple linear classifiers, more advanced methods or better features may be needed for better classification performance.

Original languageEnglish (US)
Title of host publicationUser Modeling, Adaptation, and Personalization - 18th International Conference, UMAP 2010, Proceedings
Pages135-146
Number of pages12
Volume6075 LNCS
DOIs
StatePublished - 2010
Event18th International Conference on User Modeling, Adaptation and Personalization, UMAP 2010 - Big Island, HI, United States
Duration: Jun 20 2010Jun 24 2010

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume6075 LNCS
ISSN (Print)03029743
ISSN (Electronic)16113349

Other

Other18th International Conference on User Modeling, Adaptation and Personalization, UMAP 2010
CountryUnited States
CityBig Island, HI
Period6/20/106/24/10

Fingerprint

Intelligent Tutoring Systems
Intelligent systems
Ranking
Classifiers
Classifier
Students
Model
Sensor
Baseline
Sensors
Educational Technology
Educational technology
Predict
Testing
Emotion
Generalise
Evaluate
Prediction

ASJC Scopus subject areas

  • Computer Science(all)
  • Theoretical Computer Science

Cite this

Cooper, D. G., Muldner, K., Arroyo, I., Woolf, B. P., & Burleson, W. (2010). Ranking feature sets for emotion models used in classroom based intelligent tutoring systems. In User Modeling, Adaptation, and Personalization - 18th International Conference, UMAP 2010, Proceedings (Vol. 6075 LNCS, pp. 135-146). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 6075 LNCS). https://doi.org/10.1007/978-3-642-13470-8_14

Ranking feature sets for emotion models used in classroom based intelligent tutoring systems. / Cooper, David G.; Muldner, Kasia; Arroyo, Ivon; Woolf, Beverly Park; Burleson, Winslow.

User Modeling, Adaptation, and Personalization - 18th International Conference, UMAP 2010, Proceedings. Vol. 6075 LNCS 2010. p. 135-146 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 6075 LNCS).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Cooper, DG, Muldner, K, Arroyo, I, Woolf, BP & Burleson, W 2010, Ranking feature sets for emotion models used in classroom based intelligent tutoring systems. in User Modeling, Adaptation, and Personalization - 18th International Conference, UMAP 2010, Proceedings. vol. 6075 LNCS, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 6075 LNCS, pp. 135-146, 18th International Conference on User Modeling, Adaptation and Personalization, UMAP 2010, Big Island, HI, United States, 6/20/10. https://doi.org/10.1007/978-3-642-13470-8_14
Cooper DG, Muldner K, Arroyo I, Woolf BP, Burleson W. Ranking feature sets for emotion models used in classroom based intelligent tutoring systems. In User Modeling, Adaptation, and Personalization - 18th International Conference, UMAP 2010, Proceedings. Vol. 6075 LNCS. 2010. p. 135-146. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-642-13470-8_14
Cooper, David G. ; Muldner, Kasia ; Arroyo, Ivon ; Woolf, Beverly Park ; Burleson, Winslow. / Ranking feature sets for emotion models used in classroom based intelligent tutoring systems. User Modeling, Adaptation, and Personalization - 18th International Conference, UMAP 2010, Proceedings. Vol. 6075 LNCS 2010. pp. 135-146 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).
@inproceedings{4ec6551c298f4053a3f5342cfe200118,
title = "Ranking feature sets for emotion models used in classroom based intelligent tutoring systems",
abstract = "Recent progress has been made by using sensors with Intelligent Tutoring Systems in classrooms in order to predict the affective state of students users. If tutors are able to interpret sensor data with new students based on past experience, rather than having to be individually trained, then this will enable tutor developers to evaluate various methods of adapting to each student's affective state using consistent predictions. In the past, our classifiers have predicted student emotions with an accuracy between 78{\%} and 87{\%}. However, it is still unclear which sensors are best, and the educational technology community needs to know this to develop better than baseline classifiers, e.g. ones that use only frequency of emotional occurrence to predict affective state. This paper suggests a method to clarify classifier ranking for the purpose of affective models. The method begins with a careful collection of a training and testing set, each from a separate population, and concludes with a non-parametric ranking of the trained classifiers on the testing set. We illustrate this method with classifiers trained on data collected in the Fall of 2008 and tested on data collected in the Spring of 2009. Our results show that the classifiers for some affective states are significantly better than the baseline model; a validation analysis showed that some but not all classifier rankings generalize to new settings. Overall, our analysis suggests that though there is some benefit gained from simple linear classifiers, more advanced methods or better features may be needed for better classification performance.",
author = "Cooper, {David G.} and Kasia Muldner and Ivon Arroyo and Woolf, {Beverly Park} and Winslow Burleson",
year = "2010",
doi = "10.1007/978-3-642-13470-8_14",
language = "English (US)",
isbn = "3642134696",
volume = "6075 LNCS",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
pages = "135--146",
booktitle = "User Modeling, Adaptation, and Personalization - 18th International Conference, UMAP 2010, Proceedings",

}

TY - GEN

T1 - Ranking feature sets for emotion models used in classroom based intelligent tutoring systems

AU - Cooper, David G.

AU - Muldner, Kasia

AU - Arroyo, Ivon

AU - Woolf, Beverly Park

AU - Burleson, Winslow

PY - 2010

Y1 - 2010

N2 - Recent progress has been made by using sensors with Intelligent Tutoring Systems in classrooms in order to predict the affective state of students users. If tutors are able to interpret sensor data with new students based on past experience, rather than having to be individually trained, then this will enable tutor developers to evaluate various methods of adapting to each student's affective state using consistent predictions. In the past, our classifiers have predicted student emotions with an accuracy between 78% and 87%. However, it is still unclear which sensors are best, and the educational technology community needs to know this to develop better than baseline classifiers, e.g. ones that use only frequency of emotional occurrence to predict affective state. This paper suggests a method to clarify classifier ranking for the purpose of affective models. The method begins with a careful collection of a training and testing set, each from a separate population, and concludes with a non-parametric ranking of the trained classifiers on the testing set. We illustrate this method with classifiers trained on data collected in the Fall of 2008 and tested on data collected in the Spring of 2009. Our results show that the classifiers for some affective states are significantly better than the baseline model; a validation analysis showed that some but not all classifier rankings generalize to new settings. Overall, our analysis suggests that though there is some benefit gained from simple linear classifiers, more advanced methods or better features may be needed for better classification performance.

AB - Recent progress has been made by using sensors with Intelligent Tutoring Systems in classrooms in order to predict the affective state of students users. If tutors are able to interpret sensor data with new students based on past experience, rather than having to be individually trained, then this will enable tutor developers to evaluate various methods of adapting to each student's affective state using consistent predictions. In the past, our classifiers have predicted student emotions with an accuracy between 78% and 87%. However, it is still unclear which sensors are best, and the educational technology community needs to know this to develop better than baseline classifiers, e.g. ones that use only frequency of emotional occurrence to predict affective state. This paper suggests a method to clarify classifier ranking for the purpose of affective models. The method begins with a careful collection of a training and testing set, each from a separate population, and concludes with a non-parametric ranking of the trained classifiers on the testing set. We illustrate this method with classifiers trained on data collected in the Fall of 2008 and tested on data collected in the Spring of 2009. Our results show that the classifiers for some affective states are significantly better than the baseline model; a validation analysis showed that some but not all classifier rankings generalize to new settings. Overall, our analysis suggests that though there is some benefit gained from simple linear classifiers, more advanced methods or better features may be needed for better classification performance.

UR - http://www.scopus.com/inward/record.url?scp=77954607997&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=77954607997&partnerID=8YFLogxK

U2 - 10.1007/978-3-642-13470-8_14

DO - 10.1007/978-3-642-13470-8_14

M3 - Conference contribution

SN - 3642134696

SN - 9783642134692

VL - 6075 LNCS

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 135

EP - 146

BT - User Modeling, Adaptation, and Personalization - 18th International Conference, UMAP 2010, Proceedings

ER -