Task11 @SemEval2023: Learning With Disagreements (Le-Wi-Di), 2nd edition

Organized by elisa.leonardelli - Current server time: Aug. 18, 2022, 3:44 a.m. UTC


Pre-practice Phase
July 13, 2022, midnight UTC


Practice Phase
Sept. 1, 2022, midnight UTC


Competition Ends
Download Size (mb) Phase
Public Data 0.007 #0 Pre-practice Phase

Pre-practice Phase

Start: July 13, 2022, midnight

Description: In this initial phase, you are provided with a sample of the data for each dataset (see 'Get data sample' in the Overview section). Submissions in Codalab are not allowed. If you intend to partecipate, please join also our google group so to be updated with any news about the task.

Practice Phase

Start: Sept. 1, 2022, midnight

Description: This is the practice phase of the competition. You can partecipate to Codalab and access the "Participate" and "Results" sections. In this phase, through the partecipate section, you are provided with the training and validation parts of the datasets (though labels for validation will be provided in the next phase). You are expected to craft novel approaches for training the models using the crowd labels. You may use the base models in starter_kit.zip as a starting point. Submissions are unlimited and are evaluated on the validation part. Leaderboard is public so to allow participants to see how their model performs and compare the performance with others.

Evaluation Phase

Start: Jan. 10, 2023, midnight

Description: In this phase of the competition, the test data is released. Participants are expected to use the models trained in the practice phase to make predictions on the test data (without labels) and make submissions for the test data. The number of submissions for this phase is limited to 5 submissions per participant, to prevent the participants from fine-tuning their models on the test data. However, in this phase we release the labels and crowd annotations for the development set to facilitate quick offline testing and refining of models.


Start: Feb. 1, 2023, midnight

Description: The official competition ends in the evaluation phase. However, the Post-competition phase allows you to continue to refine and test your models.

Competition Ends


You must be logged in to participate in competitions.

Sign In