Hi MAFAT Team, I have a question about reproduce of the models - in Spacenet challenges there were rule that results should be +- reproducible (but not exact match) - for example right now if I train my best config I get +- the same results but they not exact all the time (for example diff is only in third digit after the point in mAP calculations) - and when I try fixing seed in the mid training there are some problems (because I need randomness for data generation in my case) - so my question is next - do we need to be able to reproduce results exactly or some difference (lets say LB score will be 0.xy for one trained model and 0.x(y+2) for another model) is allowed?
Posted by: maksimovka @ March 19, 2023, 8:14 p.m.Hi,
The reproduced model should operate automatically on the private test set and new unseen data without significant loss of performance.
The organizer team will train several models in the validation process and will average the performance.
We assume that there is a random aspect in the creation and training of the model (although fixing "seed" where possible can come close to the submitted model), so we expect that the performance diffs won’t be significant.
MAFAT Challenge Team
Posted by: MAFAT_Challenge @ March 20, 2023, 1:28 p.m.and one more question about this part of rules - It should be mentioned that this limitation is only relevant to the inference code. Participants can use other software packages during training, as long as their inference code can be executed in the competition evaluation docker container. - does it mean also that I can train on another hardware than g4dn.xlarge - for example in my case it is A30 gpu that has more memory than T4 used in g4dn.xlarge?
Posted by: maksimovka @ March 21, 2023, 2:20 p.m.Hi,
You can train your model on your own hardware.
And upon submission on the competition platform, your inference code will be executed on the instance g4dn.xlarge, subject to the constraints detailed on the competition website.
Please note that if you need additional packages in the submission docker environment, send us a request in the Python Packages thread in the forum, and we will consider it.
Neta,
MAFAT Challenge Team