Competition

Evaluation 

Submissions are evaluated using two metrics. Focusing on subjective disagreement, the existence of a ‘truth’ cannot be assumed. To this end, a 'soft' evaluation using cross-entropy will be taken as the primary metric. A 'hard' evaluation using F1 is also performed.

1. 'Soft' evaluation: evaluates how well the model's probabilities reflect the level of agreement among annotators. It is measured using cross-entropy, as a model that correctly predicts the distribution of labels produced by the crowd for each item, will have low cross-entropy. 

2. 'Hard' evaluation: evaluates how well the results submitted align with the preferred (gold) interpretation. Each item is considered correct if it assigns the maximum probability to the preferred interpretation (if available). It is measured using micro-F1

An ideal model would have a high micro-F1 and low cross-entropy results. However, for the scope of this competition, the primary evaluation considered is the soft one (cross-entropy).

Notes:

In the starting_kit package, you can find the code used to calculate the scores and an example of the data format for submission [see more details in the "data format and data submission format section"].

Baseline: our baseline (organizers_baseline in the Results tables) is a majority baseline. The scores obtained assigning each item to the majority group (Soft label always ["0": 1.0, "1": 0.0], Hard label always "0" )

After submission, if you want, you can see your scores (see scores_correct.txt file) and show them in the "Results table" (notice that there are two separated results tables, one for the practice phase, one for the evaluation phase).

For each submission, you can decide to submit as many datasets as you wish (one file for each dataset). For the dataset(s) you don't submit, the scores of the majority baseline are assigned.

We will consider a rank with the average performances across datasets (if submissions for alla datasets are not present, we assign our baseline score). We will consider also separate rankings for each datasets. You can re-order the rankings within each for each dataset in the Results table by clicking on the arrows of the header.

Rules

1. Submissions valid for the competition must be made during the evaluation phase. You may submit a total of 5 submissions.

2. For each submission, you may submit results for any number of datasets (at least one dataset). Each dataset will have a separate leaderboard. A general leaderboard will also be created, showing the average results across datasets. In case not all the tasks are submitted, the baseline performance will be considered for the average.

4. Use a single crowd learning methodology or framework across all the tasks and datasets. For example, if your framework is to aggregate the crowd labels using Majority Voting and train on the aggregated labels, use this same methodology for all of the tasks you participate in.

5. You must incorporate the crowd labels into your training framework. 

6. You may not create multiple accounts or belong to several teams.

Rules

1. Submissions valid for the competition must be made during the evaluation phase. You may submit a total of 5 submissions.

2. For each submission, you may submit results for any number of datasets (at least one dataset). Each dataset will have a separate leaderboard. A general leaderboard will also be created, showing the average results across datasets. In case not all the tasks are submitted, the baseline performance will be considered for the average.

4. Use a single crowd learning methodology or framework across all the tasks and datasets. For example, if your framework is to aggregate the crowd labels using Majority Voting and train on the aggregated labels, use this same methodology for all of the tasks you participate in.

5. You must incorporate the crowd labels into your training framework. 

6. You may not create multiple accounts or belong to several teams.

Contacts

If you want to stay updated on the task or share information with the other participants, join our participants group.

For direct inquiries, please contact us at this address

Follow us on Twitter for any news related to learning with disagreements!

See also our webpage

Important Dates

• Sample data ready: July 15, 2022
• Training data ready: September 1, 2022
• Evaluation start: January 10, 2022
• Evaluation end: January 31, 2022
• Paper submission due: February 2023
• Notification to authors: March 2023
• Camera ready due: April 2023
• SemEval workshop: Summer 2023

All deadlines are 23:59 anywhere on earth (UTC-12).

Organisers

Elisa Leonardelli, Fondazione Bruno Kessler (FBK), Italy
Gavin Abercrombie, Heriot-Watt University, United Kingdom
Dina Almanea, Queen Mary University of London, United Kingdom
Valerio Basile, Torino University, Italy
Tommaso Fornaciari, Bocconi University, Italy
Barbara Plank, IT University of Copenhagen, Denmark
Verena Rieser, Heriot-Watt University, United Kingdom
Massimo Poesio, Queen Mary University of London, United Kingdom
Alexandra Uma, Queen Mary University of London, United Kingdom

Data format

We encourage participants in developing methods able to capture agreements/disagreements, rather than focusing on developing the best models. To facilitate participants in testing their methods across all the datasets, we developed an harmonized format across all datasets, a format that emphasize common features of datasets, while maintaining their diversities. Datasets are released in json format, below an example for each dataset is shown.

Each item of each dataset presents several fields that are common between datasets: unique_id, text, annotators, annotations, number of annotations, hard label, soft label, lang, split, other info. 

In the field "other info", the information that is dataset-specific is contained and differs from dataset to dataset. Participants can leverage on this dataset-specific information to improve performance for a specific dataset. It contains:

1. HS-Brexit dataset: the group to which annotators belong (group1: control group of other indivudals, group2: target group of Muslim immigrants in the UK). Moreover, it contains the annotations the same annotators gave for aggressiveness and offensiveness.
2. ArMIS dataset: the 'position' and gender of each annotator ('moderate female', 'liberal female', 'conservative male')
3. ConvAbuse dataset: the annotations the same annotators gave for Directedness (explicit, implicit); Target (group, individual–system, individual–3rd party); Type (general, sexist, sexual harassment, homophobic, racist, transphobic, ableist, intellectual). Annotations are binary.
4. MD-Agreement dataset: the domain of the tweet (Covid-19, Elections2020, BLM)

Notes

To note that, although the fields are the same, they contain substantially different information across datasets:

• the field text contains a tweet for the MD-Agreement, HS-Brexit and ArMIS datasets, while for ConvAbuse it contains a conversation
• The number of crowd-labels collected for each item vary across datasets: Argreement dataset is 5, HS-Brexit dataset is 6, ArMIS dataset is 3, ConvAbuse has a variable number of crowd annotations.
• The language of all datasets is English, exception for ArMIS that is Arabic. 
• The field annotations contains binary [0,1] values for all datasets, except ConvAbuse that contains "Abuse Severity" (-3,-2,-1, 0, 1). From -3 to -1 is considered abusive, while 0 to 1 is not abusive.

To note that within each dataset, annotators have been assigned and identifier starting from one (Ann1, Ann2 etc.). While within each dataset, the same annotator ID refer to the same annotator, no annotators are in common between datasets.  

To note that HS-Brexit, ArMIS and ConvAbuse datasets have been annotated by a small crowd of annotators (max 8 different annotators), conversely the MD-Agreement dataset has been annotated by a crowd of > 800 annotators (via AMT).

To note that for some datasets (ArMIS and ConvAbuse), the number of annotations collected for each item are not always odd, and majority is not always possible (e.g. soft label "0" and soft_label "1" are 0.5). In this (few) cases, the hard label has been assigned in a random manner.

The datsets:

Akhtar, S., Basile, V., & Patti, V. (2021). Whose opinions matter? perspective-aware models to identify opinions of hate speech victims in abusive language detection. arXiv preprint arXiv:2106.15896.

Almanea, D., & Poesio, M. ArMIS-The Arabic Misogyny and Sexism Corpus with Annotator Subjective Disagreements.

Leonardelli, E., Menini, S., Aprosio, A. P., Guerini, M., & Tonelli, S. (2021). Agreeing to Disagree: Annotating Offensive Language Datasets with Annotators' Disagreement. arXiv preprint arXiv:2109.13563.

Curry, A. C., Abercrombie, G., & Rieser, V. (2021). ConvAbuse: Data, analysis, and benchmarks for nuanced abuse detection in conversational AI. arXiv preprint arXiv:2109.09483.