metric calculation for this competiton is broken and apparently the order of images changes the score. ive mailed the organizers including the email marked as "urgent" on the homepage hoping to get immediate response but nada.
Posted by: abhishek @ June 14, 2024, 7:02 a.m.Hi Abhishek,
Thank you for pointing this out. It must be frustrating to encounter such an issue, especially during a challenge. Apologies for not reacting earlier. We are now mentioning on the submission webpage that the order of the submission images + predicted scores matters. It is not intuitive, but that's how the Codalab platform works. We appreciate your initiative and that you've emailed us, too.
Best regards
Posted by: vladh @ June 14, 2024, 7:28 a.m.> but that's how the Codalab platform works.
Unfortunately, that is not true. You are using a custom metric but possibly forgetting to join on image name or sort both solution and submission in the same way before calculating the scores.
Codalab allows you to do that.
Here is the example iris competition bundle that shows how to sort predictions: https://github.com/codalab/competition-examples/blob/master/codalab/Iris/iris_competition_bundle/scoring_program/score.py (you can do whatever you want!)
Posted by: abhishek @ June 14, 2024, 7:51 a.m.Thank you, Abhishek, for the suggestion. We will consider this next time.
For this challenge, please follow the rule to sort your submission according to the validation set if you wish to submit to the challenge.
Best regards,
Saman
> please follow the rule to sort your submission according to the validation set
How? in the same way as provided in sample submission? Or in a different way?
Posted by: abhishek @ June 14, 2024, 9:59 a.m.The order of images and their scores must be kept the same as provided in the validation_shared.csv document.
Posted by: stootaghaj @ June 14, 2024, 10:13 a.m.thanks
Posted by: abhishek @ June 14, 2024, 10:14 a.m.