Multilingual Tweet Intimacy Analysis

Intimacy is a fundamental social aspect of language. This SemEval shared task focuses on predicting the intimacy of tweets in 10+ languages. This task is co-organized by University of Michigan and Snap Inc.

[ALERT] You might see offensive or sexual content in the dataset.

Please email pedropei@umich.edu if you need the test set!

Full ranking and detailed scores are available in this google sheet. We encourage you to check out your detailed ranking on each language as the overall ranking may not fully reflect your effort on certain languages.  You may check our baseline performances in this task paper and please fill in this very short participation survey to access the test data and evaluation script.

Final rankings:

 Top 10 teams on predicting overall intimacy score

1.    lazybob, Ohio State University

2.    UZH_CLyp, University of Zurich

3.    opi, OPI

4.    tmn, University of Tyumen

5.    OPD, Interactive Entertainment Group of Netease Inc

6.    lottery, Individual Participant

7.    DUTH, Democritus University of Thrace

8.    Zhegu, University of Electronic Science and Technology of China

9.    arizonans, University of Arizona, University of Arizona, University of Arizona

10.    irel, International Institute of Information Technology, Hyderabad

Top 10 teams on predicting languages in the training data

1.    king001, Ping An Insurance (Group) Company

2.    lazybob, Ohio State University

3.    opi, OPI

4.    cyclejs, sunwoda

5.    OPD, Interactive Entertainment Group of Netease Inc

6.    lottery, Individual Participant

7.    ODA_SRIB, Samsung Research and Development Institute Bangalore

8.    UZH_CLyp, University of Zurich

9.    tmn, University of Tyumen

10.    water, Individual Participant

Top 10 teams on predicting languages not in the training data

1.    lazybob, Ohio State University

2.    DUTH, Democritus University of Thrace

3.    irel, International Institute of Information Technology, Hyderabad

4.    kean_nlp, Wen Zhou Kean University

5.    opi, OPI

6.    UZH_CLyp, University of Zurich

7.    Zhegu, University of Electronic Science and Technology of China

8.    arizonans, University of Arizona, University of Arizona, University of Arizona

9.    tmn, University of Tyumen

10.    ZBL2W, Dalian University of Technology

Task description

• The goal of this task is to predict the intimacy of tweets in 10 languages. You are given a set of tweets in six languages (English, Spanish, Italian, Portuguese, French, and Chinese) annotated with intimacy scores ranging from 1-5 to train your model.

• You are encouraged (but not required) to also use the question intimacy dataset (Pei and Jurgens, 2020) which contains 2247 English questions from Reddit as well as another 150 questions from Books, Movies, and Twitter. Please note that the intimacy scores in this dataset range from -1 to 1 so you might need to consider data augmentation methods or other methods mapping the intimacy scores to the 1-5 range in the current task. Please check out the paper for more details about this question intimacy dataset.

• The model performance will be evaluated on the test set in the given 6 languages as well as an external test set with 4 languages not in the training data (Hindi, Arabic, Dutch and Korean).

• We will use Pearson's r as the evaluation metric.

References

[1] Jiaxin Pei and David Jurgens. 2020. Quantifying Intimacy in Language. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5307–5326, Online. Association for Computational Linguistics.