MAFAT Satellite Vision Challenge Forum

Go back to competition Back to thread list Post in this thread

> Reproducability of baseline

Hello .Thanks for challenge and wonderful baseline.
However, after I run the baseline_model.ipynb provided at colab with 20 epoch. I could only achieve 0.6316677087 mAP. Is there anything I need to modify to get 33.8 which is the baseline score shown on the youtube video?

Posted by: baibizhe @ March 22, 2023, 10:05 a.m.

Hi,
Our baseline was trained for 10 epochs on the full training set.
Please make sure you do the same.
Neta,
MAFAT Challenge Team

Posted by: MAFAT_Challenge @ March 22, 2023, 11:10 a.m.

Thanks!
Would you mind providing the training logs of 10 epochs training on full dataset?

Best regards,
Bizhe

Posted by: baibizhe @ March 22, 2023, 12:25 p.m.

Hi,
Unfortunately, we won't be able to provide the training logs of our baseline.

MAFAT Challenge Team

Posted by: MAFAT_Challenge @ March 22, 2023, 3:25 p.m.

Hi,
I meet the same problem with you, I just train the baseline model on colab with 10 epochs and full dataset, but just got 2.1 score. You have any ideas for this problem?

Posted by: magicjimmy @ March 24, 2023, 10:55 a.m.

No. The only experiment I've done is if I load the ckpt provided , and finetune it with only 1 epoch with original colab code , I got 2.X mMP. So , I thought the traning coce might be wrong?

Posted by: baibizhe @ March 24, 2023, 3:39 p.m.

Hi baibizhe,
I recommend ensuring the followings:
1. You have trained your model on the full dataset, not the mini dataset (notice the dataset selection at the beginning of the baseline notebook).
2. You loaded the checkpoint provided in the sample submission section in the provided scripts (notice at the baseline notebook, in "Download pre-trained weights and config" section, the pretrianed model is on DOTA dataset)
Neta,
MAFAT Challenge Team

Posted by: MAFAT_Challenge @ March 26, 2023, 10:13 a.m.

Hi Neta,
Thanks for your reply, I ensured that I have done 2 things you were commented, first I changed mini data to full data, and then run all cell, I saw that the notebook download pretrain weight and use it to train, and the evaluate results are about 95 mAP, and around 0.01 loss. That's why I wonder why when submitting the results, it doesn't detect anything and only has about 2.1 score.

Posted by: magicjimmy @ March 27, 2023, 2:48 a.m.
Post in this thread