Just submitted my first scores using the baseline code. Somehow my baseline my baseline score < leaderboard baseline score which I assumed uses the same code. Is there any difference in the code used for the "leaderboard baseline score" compared to the MNTD baseline code from Github? more epochs ? just a diffeent random seed? something else?
Posted by: ildefons @ Aug. 10, 2022, 8:25 p.m.Hello,
The MNTD baseline uses the same code in example_submission.ipynb, except that 100% of the networks are used for training instead of only 80%. In the example submission notebook, we split off 20% of the training data to use as an offline validation set. You can adjust this by changing the fraction in "split = int(len(dataset) * 0.8)". This should give you similar numbers to the baseline in the leaderboard, although let us know if this isn't the case! There is also some variation across runs due to random initialization, but it shouldn't be more than a few percentage points.
All the best,
Mantas (TDC co-organizer)
Hello,
I modified the code as required to use 100% of training data to train the model (i.e. split = len(dataset)#int(len(dataset) * 0.8) )
But the performance barely improved and is stuck at 0.46:
(previous submission)3 46.13 mntd_submission.zip 08/11/2022 08:55:16 9150 Finished
(new submission)4 46.4848 mntd_submission.zip 08/11/2022 17:45:31 9298 Finished
There must be something else. What can it be?
Posted by: ildefons @ Aug. 11, 2022, 5:56 p.m.Hello,
Which version of PyTorch are you using? We recommend using PyTorch 1.11.0, and other participants have had problems when using other versions. Also, are you using the updated version of the validation set? We switched to a new validation set for the detection track on July 25 due to a loophole in the old version.
All the best,
Mantas (TDC co-organizer)
I start using the new validation dataset and performance is now Ok
Thank you!