The primary evaluation metric will be F1-score and we also plan to report Recall.

Submission Format

The competition will take place on CODALAB.

This year we will also require participants to fill in a form describing their model and data choices for each submission.

For each submission you wish to make (under “Participate>Submit” on codalab), please upload a single zip file with the predictions and the system metadata.

For the metadata, we expect a ‘metadata.txt’ file, with exactly two non-empty lines which are for the teamname and the short system description, respectively. The first line of metadata.txt must contain your team name. You can use your CodaLab username as your teamname. The second line of metadata.txt must contain a short description (2-3 sentences) of the system you used to generate the results. This description will not be shown to other participants. Note that submissions without a description will be invalid. It is fine to use the same submission for multiple submissions/phases if you use the same model (e.g. a multilingual or multitasking model)

For the predictions we describe the exact format expected below:

Line 1: <DISK FOOTRPINT (in bytes, without compression)>

Line 2: <NUMBER OF PARAMETERS>

Line 3: <NUMBER OF ENSEMBLED MODELS> (set to 1 if there is no ensemble)

Lines 4-n where -n is the number of test samples: <LANGUAGE PAIR> <METHOD NAME> <SEGMENT NUMBER> <TARGET SENTENCE> <ERROR START INDICES> <ERROR END INDICES> <ERROR TYPES>

Where:

• LANGUAGE PAIR is the ID (e.g. en-de) of the language pair of the plain text translation file you are scoring. Follow the LP naming convention provided in the test set.
• METHOD NAME is the name of your quality estimation method.
• SEGMENT NUMBER is the line number of the plain text translation file you are scoring (starting at 0).
• TARGET SENTENCE is the target sentence based on which the error span indices were extracted. You should use exactly the target sentence as provided by the test set to ensure alignment with the gold labels.
• ERROR START INDICES the start indices (character level) of every exrror span extracted. For multiple error spans separate indices by a whitespace. For no errors output -1.
• ERROR END INDICES the end indices (character level) of every exrror span extracted. For multiple error spans separate indices by a whitespace. For no errors output -1.
• ERROR TYPES indication of minor or major error for each detected error span. The number of indices should match the number of errors. If there is no error span in a segment indicate with no-error.

Each field should be delimited by a single tab (<\t>) character.

Output example

2409244995
2280000000
3
he-en <\t> example-ensemble <\t> 0 <\t> This is a sample translation without errors. <\t> -1 <\t> -1 <\t> no-error
he-en <\t> example-ensemble <\t> 1 <\t> This is a sample translation with a span that is considered major error and another span that is considered minor error. <\t> 49 97 <\t> 70 118 <\t> major minor …