Hello,
Thank you very much for your work and the organization of the competition. For reproducibility and analysis purposes, we would like to kindly ask you for the evaluation script of the task, as we are producing a more in-depth analysis of our systems. With our script, we are not able to reproduce the exact performance numbers.
Thank you in advance,
Claudiu Daniel Hromei - extremITA
Dear participant. Thanks very much for your kind words. Please, you can refer to the google collabs examples (in the Participate tab section) in order to know how the evaluation scripts are. It this ok for you?
Kind regards
Posted by: JoseAGD @ June 23, 2023, 9:26 a.m.Thank you for your quick response, but what we would need is a script that takes the submission file, the gold standard file and computes the performance metrics, as this kind of script is provided by other tasks in the EVALITA challenge. Is it possible to have such an evaluation script?
Thank you again.
Posted by: chromei @ June 23, 2023, 9:38 a.m.Dear user. I have checked with my colleagues but we can not release the evaluation script right now. Anyway, as commented earlier, the script is the same that you can find in the first google colab, and the gold labels are available under the Files tab. You can download them and test that the results should be the same calculating the macro F1-score
Kind regards
Posted by: JoseAGD @ June 24, 2023, 11:05 a.m.