Dear organizers,
I faced the issue mentioned in the title.
I am trying to run some codes, but if I try to install "george", which is in the provided bomo.py, and "ConfigSpace", the runtime exceeds the time limit.
Since I do not get any helpful error logs, I cannot even debug it.
Do you have any ideas why this happens?
Sincerely yours,
Shuhei
Posted by: nabenabe0928 @ May 20, 2022, 6:11 a.m.Another question is:
Is there any runtime limit for this competition?
It seems my previous submissions failed after approx. 30min.
Does that mean we need to finish 3 settings x 10 runs during the final phase in 30 min?
Or will not there be any time limit during the final phase and we have it only for the development phase?
Sincerely,
Shuhei
Posted by: nabenabe0928 @ May 20, 2022, 12:21 p.m.Hi Shuhei,
Regarding the runtime limit, it was 25min but I've now set it to 2 hours. From your "prediction_stderr_file.txt" file, it does seem like you got the error "Execution time limit exceeded." You're right that the runtime limit is the total time for all 10 runs x 2 settings (in Dev phase) and 10 runs x 3 settings (in Final phase). We don't intend for the runtime limit to be a constraint on participant's algorithm in this evaluation, so I am happy to increase it more if needed.
Regarding potential issues with modules not installing, we'll need to look into this deeper. Can you first confirm if it is still a problem? Let me know which submission ID is this and we can figure it out. Try to put more print statements in each step of your code if possible, since CodaLab doesn't expose a lot of information to us inherently.
Posted by: kevinduh @ May 20, 2022, 3:49 p.m.Hi Kevin,
Thanks for your response.
Regarding the runtime limit, 2h might be troublesome for people (for now) as we just get a failure after 2h of submission in case we got the module install errors.
About the module errors submissions, they are submitted at
05/20/2022 05:10:10 and 05/20/2022 05:28:27.
I cannot check the error immediately, but the module installations are definitely the cause as I modified codes so that the module installations are called in the very beginning of the run.
More specifically, the installations are not in each model file, but it's at the head of hpo_model.py.
And the error files stopped during one of the module installations.
So the error seems to be happening in system call, but not in Python Scripts.
That's why I cannot print any further information...
I think the runtime limit and module installation error can be solved separately.
For runtime limit, the 2hr limit can be increased if needed. If the code fails before that for another reason, I believe you will see that in the codalab web interface before the 2hr is up. So hopefully that problem is resolved for now.
For the module installation error, someone else had a similar problem. I've just increased the RAM of the compute worker. Please resubmit and let me know if that works. If not, then let's dig further.
Posted by: kevinduh @ May 20, 2022, 7:11 p.m.