Hi, I submitted my work for testing the sub-dataset result yesterday but still didn't get the evaluation result.
Can you help me find out what goes wrong?
Hi Chih-Hai_Su,
Please download the starting kit from the following link:
https://codalab.lisn.upsaclay.fr/competitions/13273#learn_the_details-get_starting_kit
The provided python script, named 6_examine_submission_file.py, will tell you what caused the failure of the submission in a second.
Posted by: youngkyoonjang @ July 7, 2023, 9:49 a.m.Hi Chih-Hai Su,
Normally, it doesn’t take longer than 10 minutes to get results when a submission is successful. Can you please double-check how I created PNG files using the PIL library in my starting kit (4_create_dummy_submission_zip.py)? I can see that you submitted more than 5 hours ago, but the status is still showing as “running”. This has never happened with other successful submissions. Honestly, I’m not sure what’s wrong if the examine script doesn’t show any errors from the submission zip file. When I noticed that your submission was still “running” after several hours, I had to confirm that it shouldn’t take this long.
Posted by: youngkyoonjang @ July 7, 2023, 5:02 p.m.Dear organizers,
Thanks for your advice.
I followed the 4_create_dummy_submission_zip.py to generate all my submission file, and I directly submitted it to the system.
Though it shows 0.0 in the score, the status still shows as "running".
By the way, there's "WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap." showing in the std_err.txt.
Someone who met with this error has submission fail, too. (https://github.com/codalab/codalab-competitions/issues/2490)
Hi Chih-Hai Su,
We are aware of the warning message. However, it has never blocked submissions made by other participants, including yours. I have tested submitting your file after downloading it, but ironically, it has never been successful, while other zip files have. At least, you should be able to submit the dummy_submission.zip file, which is automatically created by our starting kit python script and submitting the dummy_submission.zip file has never been failed. I can confirm that even between your multiple submissions, there were several successful submissions made by different participants. Unfortunately, we don’t plan to test results for individual submissions. I hope you can find a solution.
Posted by: youngkyoonjang @ July 7, 2023, 11:05 p.m.Hi organizers,
Can you help me to kill the process? I want to test the dummy_submission.zip first.
Hi Chih-Hai Su,
Although there are submissions that are held in ‘running’ status, you can still submit other files as separate results, which will be independently successful. I have deleted your submissions that were held in ‘running’ status, but please bear in mind that your submissions held in ‘running’ status do not block your next successful submission.
Posted by: youngkyoonjang @ July 8, 2023, 8:33 a.m.Hi organizers,
It seems the problem is solved, though I don't know why.
Thanks for your help.
Hello organizers,
I am also encountering the same problem, my submissions are stuck in the "Running" stage. However, I received the evaluation score on my submission page, but it doesn't get updated in the leaderboard.
Could you please share some insights on why this might be happening; is it possible that the submission size is too large (196,2MB)?
Thank you!
Posted by: andreead @ July 12, 2023, 8:09 a.m.Hi andreead,
As the issue occurs when there is an increased volume of submissions on the same day, I suspect that the CodaLab system couldn’t handle it properly. However, I can confirm that the size of 196.2 MB is not a problem. In this case, please self-evaluate your submission using your evaluation score on your submission page and compare your score with others on the leaderboard as a feedback for your development.
I will work on figuring out how to avoid this for now. Even though it appears to be problematic, I believe the system should work properly during the Final Challenge Phase. We only allow a limited number (3) of submissions per day during the Final Challenge Phase, so I expect there will be no problem with Challenge Phase submissions. We want to reassure you that just in case it happens during the Challenge Phase, we will handle it by downloading your original submission for evaluation or by finding an alternative way to rank properly.
Posted by: youngkyoonjang @ July 12, 2023, 11:13 a.m.