Hi, will the development phase score be counted in the final competition result?
How can I choose my best model as the submitted result? Now, only the last submitted result performance will be shown on the results leaderboard.
Dear participant,
no, the development score will not be used to define the top ranked methods. This will be based on the results obtained at the test phase.
Regarding the other question, Codalab will keep on the leaderboard the last valid submission. This helps participants to receive real-time feedback on the submitted files. Participants are responsible to upload the file they believe will rank them in a better position as a last and valid submission. Note that at the test phase you only have a max number of submissions = 3. This is a conservative number to take into account possible failures, but mainly to avoid participants to improve the score at the test phase by trial and error.
Best
Thank you for your explanation!
Posted by: yjhuang @ June 20, 2022, 5:57 a.m.