Forums

PRINCE Out-of-distribution Generalization Challenge @ ECML-PKDD Forum

Go back to competition Back to thread list Post in this thread

> how to select additional final subs?

Hi,

Thanks for sponsoring such an interesting challenge.

How do we select the additional final subs? And is the pre-selected one the one that had the highest score on the never seen LB environment?

Thanks

Posted by: mariofilho @ May 30, 2022, 12:11 p.m.

Your best solution on phase 1 (Dev) as per the NCE on the Never Seen Env is automatically selected for Final phase.

You can post 2 additional submissions in final phase.

In case you want to select manually one of them you need to use the button "Submit to Leaderboard" in the details of you submission (click the cross).

HTH

Posted by: eustache @ May 31, 2022, 6:21 a.m.

Dear,

I can't seem to find a "submit to leaderboard" button in the final phase. It seems like a subset of my submissions from the dev phase have been added there, but I do not see how I can select any of these as a "final submission".
I also have another question: How should we go about selecting submissions when part of a team? Can each member select 1 + 2 final submissions (best + 2 chosen) or can we only submit a total of three submissions? If the latter is the case, are we obliged to select the best solution from both team members and then one additional submission, or can one member select all three submissions and the other one none?
Thanks in advance!

Posted by: smortier @ May 31, 2022, 7:16 a.m.

Thanks for the answer.

Like @smortier said, the option to submit to the leaderboard is not available for the current submissions and when I try to re-submit a solution it says: "The maximum number of submissions has been reached"

Maybe it's a bug?

Posted by: mariofilho @ May 31, 2022, 12:11 p.m.

Re: final submissions, let me check the configuration and do some manual test on a copy of the competition to make sure of what is happening exactly.

Re: teams you shall make at most 2 submissions for the whole team in Final phase.

Sorry for the inconvenience, will let you know as soon as we have a solid answer.

Eustache

Posted by: eustache @ May 31, 2022, 4:20 p.m.

Thank you for your patience. After careful inspection we have found the issue - it is a specific behavior of the Codalab platform related to auto-migration.

I'm sending now instructions to all participants wrt to the final phase and how to submit your additional 2 submissions.

Please post here when you have sent an additional submission so that we know that it worked for you.

Posted by: eustache @ June 1, 2022, 1:27 p.m.

Hi Eustache,

I just re-submitted two solutions and it worked!

Thank you!

Posted by: mariofilho @ June 1, 2022, 3:19 p.m.

Hi,

I just submitted one submission to the final phase. However, even before that, there were already 13 submissions in that list (automatically cloned from dev phase). Can you confirm that I successfully submitted my final submission? (teammate will submit the other one and my best one should be automatically cloned from the dev phase)

Thanks,
smortier

Posted by: smortier @ June 1, 2022, 4 p.m.

@smortier: yes I can see your submission. The submission count encompasses all auto-migrated solutions so it is expected to be > 2.

Posted by: eustache @ June 2, 2022, 6:40 a.m.

Hi,

Wanted to know if one doesn't submit now in Final phase, his/her best solution from development phase will automatically be considered or not? Or should we resubmit the best one from development phase again manually out of the 2 submissions that are provided now in Final phase?

Thanks.

Posted by: ColdWheels @ June 2, 2022, 7:35 a.m.

@ColdWheels: no need to resubmit anything if you are happy with your best dev phase solution - it is included by default in the final leaderboard.

Posted by: eustache @ June 2, 2022, 8:14 a.m.
Post in this thread