Classification of closely related sub-dialects of Arabic using support-vector machines

Samantha Wray

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Abstract

    Colloquial dialects of Arabic can be roughly categorized into five groups based on relatedness and geographic location (Egyptian, North African/Maghrebi, Gulf, Iraqi, and Levantine), but given that all dialects utilize much of the same writing system and share overlapping features and vocabulary, dialect identification and text classification is no trivial task. Furthermore, text classification by dialect is often performed at a coarse-grained level into these five groups or a subset thereof, and there is little work on sub-dialectal classification. The current study utilizes an n-gram based SVM to classify on a fine-grained sub-dialectal level, and compares it to methods used in dialect classification such as vocabulary pruning of shared items across dialects. A test case of the dialect Levantine is presented here, and results of 65% accuracy on a four-way classification experiment to sub-dialects of Levantine (Jordanian, Lebanese, Palestinian and Syrian) are presented and discussed. This paper also examines the possibility of leveraging existing mixed-dialectal resources to determine their sub-dialectal makeup by automatic classification.

    Original languageEnglish (US)
    Title of host publicationLREC 2018 - 11th International Conference on Language Resources and Evaluation
    EditorsHitoshi Isahara, Bente Maegaard, Stelios Piperidis, Christopher Cieri, Thierry Declerck, Koiti Hasida, Helene Mazo, Khalid Choukri, Sara Goggi, Joseph Mariani, Asuncion Moreno, Nicoletta Calzolari, Jan Odijk, Takenobu Tokunaga
    PublisherEuropean Language Resources Association (ELRA)
    Pages3671-3674
    Number of pages4
    ISBN (Electronic)9791095546009
    StatePublished - Jan 1 2019
    Event11th International Conference on Language Resources and Evaluation, LREC 2018 - Miyazaki, Japan
    Duration: May 7 2018May 12 2018

    Publication series

    NameLREC 2018 - 11th International Conference on Language Resources and Evaluation

    Other

    Other11th International Conference on Language Resources and Evaluation, LREC 2018
    CountryJapan
    CityMiyazaki
    Period5/7/185/12/18

    Fingerprint

    dialect
    vocabulary
    colloquial
    Support Vector Machine
    Sub-dialects
    Group
    experiment
    resources

    Keywords

    • Language identification
    • Text classification
    • Validation of language resources

    ASJC Scopus subject areas

    • Linguistics and Language
    • Education
    • Library and Information Sciences
    • Language and Linguistics

    Cite this

    Wray, S. (2019). Classification of closely related sub-dialects of Arabic using support-vector machines. In H. Isahara, B. Maegaard, S. Piperidis, C. Cieri, T. Declerck, K. Hasida, H. Mazo, K. Choukri, S. Goggi, J. Mariani, A. Moreno, N. Calzolari, J. Odijk, ... T. Tokunaga (Eds.), LREC 2018 - 11th International Conference on Language Resources and Evaluation (pp. 3671-3674). (LREC 2018 - 11th International Conference on Language Resources and Evaluation). European Language Resources Association (ELRA).

    Classification of closely related sub-dialects of Arabic using support-vector machines. / Wray, Samantha.

    LREC 2018 - 11th International Conference on Language Resources and Evaluation. ed. / Hitoshi Isahara; Bente Maegaard; Stelios Piperidis; Christopher Cieri; Thierry Declerck; Koiti Hasida; Helene Mazo; Khalid Choukri; Sara Goggi; Joseph Mariani; Asuncion Moreno; Nicoletta Calzolari; Jan Odijk; Takenobu Tokunaga. European Language Resources Association (ELRA), 2019. p. 3671-3674 (LREC 2018 - 11th International Conference on Language Resources and Evaluation).

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Wray, S 2019, Classification of closely related sub-dialects of Arabic using support-vector machines. in H Isahara, B Maegaard, S Piperidis, C Cieri, T Declerck, K Hasida, H Mazo, K Choukri, S Goggi, J Mariani, A Moreno, N Calzolari, J Odijk & T Tokunaga (eds), LREC 2018 - 11th International Conference on Language Resources and Evaluation. LREC 2018 - 11th International Conference on Language Resources and Evaluation, European Language Resources Association (ELRA), pp. 3671-3674, 11th International Conference on Language Resources and Evaluation, LREC 2018, Miyazaki, Japan, 5/7/18.
    Wray S. Classification of closely related sub-dialects of Arabic using support-vector machines. In Isahara H, Maegaard B, Piperidis S, Cieri C, Declerck T, Hasida K, Mazo H, Choukri K, Goggi S, Mariani J, Moreno A, Calzolari N, Odijk J, Tokunaga T, editors, LREC 2018 - 11th International Conference on Language Resources and Evaluation. European Language Resources Association (ELRA). 2019. p. 3671-3674. (LREC 2018 - 11th International Conference on Language Resources and Evaluation).
    Wray, Samantha. / Classification of closely related sub-dialects of Arabic using support-vector machines. LREC 2018 - 11th International Conference on Language Resources and Evaluation. editor / Hitoshi Isahara ; Bente Maegaard ; Stelios Piperidis ; Christopher Cieri ; Thierry Declerck ; Koiti Hasida ; Helene Mazo ; Khalid Choukri ; Sara Goggi ; Joseph Mariani ; Asuncion Moreno ; Nicoletta Calzolari ; Jan Odijk ; Takenobu Tokunaga. European Language Resources Association (ELRA), 2019. pp. 3671-3674 (LREC 2018 - 11th International Conference on Language Resources and Evaluation).
    @inproceedings{63857a4ff5f048e1853a1e9833ffb2dc,
    title = "Classification of closely related sub-dialects of Arabic using support-vector machines",
    abstract = "Colloquial dialects of Arabic can be roughly categorized into five groups based on relatedness and geographic location (Egyptian, North African/Maghrebi, Gulf, Iraqi, and Levantine), but given that all dialects utilize much of the same writing system and share overlapping features and vocabulary, dialect identification and text classification is no trivial task. Furthermore, text classification by dialect is often performed at a coarse-grained level into these five groups or a subset thereof, and there is little work on sub-dialectal classification. The current study utilizes an n-gram based SVM to classify on a fine-grained sub-dialectal level, and compares it to methods used in dialect classification such as vocabulary pruning of shared items across dialects. A test case of the dialect Levantine is presented here, and results of 65{\%} accuracy on a four-way classification experiment to sub-dialects of Levantine (Jordanian, Lebanese, Palestinian and Syrian) are presented and discussed. This paper also examines the possibility of leveraging existing mixed-dialectal resources to determine their sub-dialectal makeup by automatic classification.",
    keywords = "Language identification, Text classification, Validation of language resources",
    author = "Samantha Wray",
    year = "2019",
    month = "1",
    day = "1",
    language = "English (US)",
    series = "LREC 2018 - 11th International Conference on Language Resources and Evaluation",
    publisher = "European Language Resources Association (ELRA)",
    pages = "3671--3674",
    editor = "Hitoshi Isahara and Bente Maegaard and Stelios Piperidis and Christopher Cieri and Thierry Declerck and Koiti Hasida and Helene Mazo and Khalid Choukri and Sara Goggi and Joseph Mariani and Asuncion Moreno and Nicoletta Calzolari and Jan Odijk and Takenobu Tokunaga",
    booktitle = "LREC 2018 - 11th International Conference on Language Resources and Evaluation",

    }

    TY - GEN

    T1 - Classification of closely related sub-dialects of Arabic using support-vector machines

    AU - Wray, Samantha

    PY - 2019/1/1

    Y1 - 2019/1/1

    N2 - Colloquial dialects of Arabic can be roughly categorized into five groups based on relatedness and geographic location (Egyptian, North African/Maghrebi, Gulf, Iraqi, and Levantine), but given that all dialects utilize much of the same writing system and share overlapping features and vocabulary, dialect identification and text classification is no trivial task. Furthermore, text classification by dialect is often performed at a coarse-grained level into these five groups or a subset thereof, and there is little work on sub-dialectal classification. The current study utilizes an n-gram based SVM to classify on a fine-grained sub-dialectal level, and compares it to methods used in dialect classification such as vocabulary pruning of shared items across dialects. A test case of the dialect Levantine is presented here, and results of 65% accuracy on a four-way classification experiment to sub-dialects of Levantine (Jordanian, Lebanese, Palestinian and Syrian) are presented and discussed. This paper also examines the possibility of leveraging existing mixed-dialectal resources to determine their sub-dialectal makeup by automatic classification.

    AB - Colloquial dialects of Arabic can be roughly categorized into five groups based on relatedness and geographic location (Egyptian, North African/Maghrebi, Gulf, Iraqi, and Levantine), but given that all dialects utilize much of the same writing system and share overlapping features and vocabulary, dialect identification and text classification is no trivial task. Furthermore, text classification by dialect is often performed at a coarse-grained level into these five groups or a subset thereof, and there is little work on sub-dialectal classification. The current study utilizes an n-gram based SVM to classify on a fine-grained sub-dialectal level, and compares it to methods used in dialect classification such as vocabulary pruning of shared items across dialects. A test case of the dialect Levantine is presented here, and results of 65% accuracy on a four-way classification experiment to sub-dialects of Levantine (Jordanian, Lebanese, Palestinian and Syrian) are presented and discussed. This paper also examines the possibility of leveraging existing mixed-dialectal resources to determine their sub-dialectal makeup by automatic classification.

    KW - Language identification

    KW - Text classification

    KW - Validation of language resources

    UR - http://www.scopus.com/inward/record.url?scp=85059891341&partnerID=8YFLogxK

    UR - http://www.scopus.com/inward/citedby.url?scp=85059891341&partnerID=8YFLogxK

    M3 - Conference contribution

    T3 - LREC 2018 - 11th International Conference on Language Resources and Evaluation

    SP - 3671

    EP - 3674

    BT - LREC 2018 - 11th International Conference on Language Resources and Evaluation

    A2 - Isahara, Hitoshi

    A2 - Maegaard, Bente

    A2 - Piperidis, Stelios

    A2 - Cieri, Christopher

    A2 - Declerck, Thierry

    A2 - Hasida, Koiti

    A2 - Mazo, Helene

    A2 - Choukri, Khalid

    A2 - Goggi, Sara

    A2 - Mariani, Joseph

    A2 - Moreno, Asuncion

    A2 - Calzolari, Nicoletta

    A2 - Odijk, Jan

    A2 - Tokunaga, Takenobu

    PB - European Language Resources Association (ELRA)

    ER -