Competition

Evaluation Setting

To allow the individual evaluation of normalization and classification systems, the evaluation will be divided in two phases:

• Phase 1: Sub-task 1 (NER) + Sub-task 4 (End-to-End)
• Phase 2: Sub-task 2 (Normalization) + Sub-task 3 (Classification)

Because of this, the test data will be released in two parts. For Phase 1, only the test text files will be released. Once it’s finished, the list of entities found in the test files will be published so that participants can create predictions for their normalization and classification in Phase 2. Participants are allowed to re-use their systems for Sub-task 4 in the rest of the tasks.

The schedule for these phases is available on https://temu.bsc.es/meddoplace/schedule/.

You will find that in this CodaLab there is a Practice and Final submission for each sub-task. Use the Practice submission to check that everything works correctly and to evaluate your system in a reduced version of the test set. Once you feel you are ready, use the Final submission to upload the predictions that you want to count towards the task, which will be evaluated on the complete test set. If you need to do some evaluations on your own, you can also use the MEDDOPLACE scorer.

For more information on the evaluation and sub-tasks, including data formats and tips, please read the Task Guide.

Evaluation Metrics

Tasks 1 and 4 will use as their main metric strict, micro-averaged precision, recall and F-1 score, while Tasks 2 and 3 will use accuracy (percentage of correct mentions out of the total). In addition, some sub-tasks include additional metrics that may help further understand and interpret the results:

The additional metrics given for the GeoNames and PlusCodes normalizations are distance-based metrics, which are more appropriate for Toponym Resolution. They are described in:

Gritta, M., Pilehvar, M.T. & Collier, N. A pragmatic guide to geoparsing evaluation. Lang Resources & Evaluation 54, 683–712 (2020). https://doi.org/10.1007/s10579-019-09475-3 [pages 694-697]

Submission Format

Predictions must be submitted as .TSV files with one annotation per row (same as the training data format). You must include headers as the first row. These are the columns for each sub-task (provided fields are in round letters, while the fields that participants must predict are in italics):

• Sub-task 1 (NER): filename, label, start_span, end_span and text
• Sub-task 2 (Normalization): filename, label, start_span, end_span, text, normalization and source
• Sub-task 3 (Classification): filename, label, start_span, end_span, text and class
• Sub-task 4 (End-to-end): filename, label, start_span, end_span, text, normalization, source and class

There is more information on format specifics and the meaning of each of the columns in the Task Guide, check it out here.

With your Final submissions, you also need to add a .TXT file that includes:

• Your team's name (e.g."teambsc") and your contact information.
• Short system descriptions (e.g. "Spanish Roberta model fine-tuned on...").
• Used resources (e.g. MEDDOPLACE corpus, gazetteer obtained from...).

Put together in a .ZIP file one .TSV file (and one .TXT) per submission and give it a recognizable name (e.g. your team's name + a short description + a number; teambsc_roberta_01.zip). It is important that there is only one .TSV file inside your .ZIP file, otherwise the scorer will fail.

MEDDOPLACE Terms and Conditions