My question is about a concern i have, regarding the the Final Phase allowed running time.
In the Final Phase description it says the allowed running time will be 9 hours since the meta-test tasks will be 6000 any-way any-shot tasks. In the Feedback Phase we have 5 hours for 1000 any-way any-shot tasks. Since the meta test tasks is 6 times more, then the time also should be x6 and thus a viable execution time would be 30 hours.
This means that some of the valid submissions which are presented to the Leaderboard of the Feedback phase will fail into the final phase.
Do i miss something?
Thank you very much.
Posted by: EmmanuelPintelas @ July 30, 2022, 2:03 p.m.Dear participant,
Indeed, as you mentioned, we will have 6 times more meta-test tasks during the final phase than during the feedback phase, but the computing time is not only used for meta-testing but also for meta-training. The idea is to have the same time for meta-training in both phases and include extra time just for meta-testing. Therefore, we checked the time our provided baselines take during meta-testing, and we found that the train-from-scratch is the method that takes the most time to process each task at meta-test time. It takes around 2.3 seconds per task (approximately four times more than Prototypical Networks and two times more than MAML). Thus, during the feedback phase, it will take around 38 minutes (2.3 seconds x 1000 meta-test tasks), and during the final phase, it will take about 3.8 hours (2.3 seconds x 6000 meta-test tasks). If we compare both times, there is a difference of approximately 3.2 hours. For this reason, we incremented 4 hours during the final phase. In a nutshell, we only increment the meta-test time, not the meta-training time.
Please, do not hesitate to contact us in case of any further questions.
Best regards,
Dustin Carrión
(for the MetaDL challenge organizing team)
Yes, but a submission examble may be like: 1hour for meta-train and 3 hours for meta-test (in feedback phase). this is a valid submission for the feedback phase (4hours total < 5) but it would fail in the final phase since it would need: 3x6=18 hours for meta-testing.
The worst case scenario would be a submission like: 0 for meta train and 5 for meta test and thus it would need 5x6=30 hours.
This means that possible valid submissions in the feedback phase would not be valid in the final phase.
I suggest that either the final phase time has to be increased into the worst case scenario (30 h) in order to be guarandeed that all valid submissions in the feedback phase to be also valid for the final, or either the meta testing time has to have a seperate max time e.g. 30 min (but such restriction would limit the inovation of solutions during the meta test time).
Thank you very much.
Posted by: EmmanuelPintelas @ Aug. 1, 2022, 10:28 a.m.My submission (76% accuracy, which is top in the leaderboard), is around 1hour meta-train and 2 hours meta-test = 3 hours in total. Will it be also valid in the final phase? That’s my concern.
Posted by: EmmanuelPintelas @ Aug. 2, 2022, 6:26 p.m.