Competition

Welcome to SemEval2022 Task3-Subtask2 PreTENS!

PreTENS is articulated into the two following sub-tasks:

• a binary classification sub-task (SubTask1), which consists in predicting the acceptability label assigned to each sentence of the test set;

• a regression sub-task (SubTask2), which consists in predicting the average score assigned by human annotators on a seven point Likert-scale with respect to the subset of data evaluated via crowdsourcing.

Task Website: https://sites.google.com/view/semeval2022-pretens/

Git Repository: https://github.com/shammur/SemEval2022Task3

This is the submission system for Subtask2. 

The evaluation phase will run from 18th January till 22nd (AOE) January 2022. The rules of the evaluation are given below:

Each subtask includes test data in three languages: English, French and Italian.

For each sub-task, the ranking will be two fold:

1. Ranking per language (En, Fr, It)

2. Global ranking - the average of the score from all the three languages (This will be the official ranking).

Each team can submit at max 3 the results of three runs to be evaluated. The evaluation phase will be active for 5 days, then the portal will be closed. For the official ranking, we will use Spearman correlation, Rho (subtask 2).

Rules: 

Submission Format:

The participant team will submit a zip file containing the prediction tsv (named as answer.tsv [MUST!!]), the naming convention of the zip file will be as follows:

teamName_subtask2_submissionNo.zip

Containing → answer.tsv file (this file will include all/chosen language predictions.

The submitted file will/can contain instances from (all)  the languages. For example: if a team (team A) chooses to participate in both English and French-Subtask 2, then the submitted prediction file will contain instances from both languages. In this case, we will choose the global ranking score (GRS) as the best in all 3 languages.

So in this case, for team A, GRS will be calculated using Rho for English and French, while for Italian (not done) the Rho will be set to 0 (However, at the competition end we will re-evaluate the ranking manually).

The format of the answer.tsv file will be as follows:

#Tab separated: ID\tLabels

answer.tsv:

format: ID\tScore

ID Score

en_261 1.09

en_440 5.64

fr_19 6

fr_254 6.67

it_21 4.42

it_191 1.33

In addition, the participant has to fill up the information below. Any submission without the information below will be invalid for the evaluation phase. 

The information need in the Description Section:

Organization: 

Team Name (mandatory): 

Team Members: None/ members list

Method Name: 

Method Description (Optional): 

Contact Email (mandatory): 

Submission per team will be accepted from one user account only.

Quick Recap

1. Maximum Submission : 3 result submissions per subtask

2. Ranking: Two ranking per subtask - Per Language Ranking and Global Ranking

3. What results will be displayed/used in the LeaderBoard: Rho (per language with a global score) for subtask2 will be shown.

4. The naming convention for submission file: The result/submission file will be tab separated (with headers: ID \t Labels/Score), named as answer.tsv and then compressed to a zip file with naming convention: <teamName_subtaskX_submissionNo.zip>, X={1,2} and No={1,2,3}

5. Results selected to display in Leaderboard: Each team will have 3 chances (per task) and from there they can choose which results to submit in the leaderboard. However, each team must submit at least one result in the board (they can change the selected entry to show anytime during competition). This is mainly given so participants attempting just selected language are not penalized by the global-ranking score mechanism.