Thank you for organizing this challenge, and we have some questions about the Final Code Submission phase:
1. How to rank teams by criteria?
The main criteria contain SROCC, PLCC, RMSE, and Runtime Complexity, introduced on the challenge's homepage. However, we find nothing about how to rank teams by these criteria. In general, there is a trade-off between performance and complexity, so a clear illustration may be helpful to us for a workable competition.
2. How long will it take to get the final result?
The GitHub repo of the challenge tells us that we should submit only one model in the Final Code Submission phase, and only the last of multiple submissions will be available. But how long will it take to get the final result (e.g., the performance of our model), after the end of this challenge or several hours later of submission?
Hi darkness,
Thanks for reaching out.
1. How to rank teams by criteria?
We are in the discussion of the final criteria. It will be finalized by the end of this week and will be published on our website.
You can definitely focus on the performance first. SROCC will be the main metric we used. PLCC and RMSE after non-linear transform might not be used.
We're considering giving the 3rd prize to the fastest model which shows a reasonable performance compared with 1st and 2nd in SROCC. But it's not finalized yet. You can focus on SROCC during development.
2. How long will it take to get the final result?
After the code submission closed on 12/15/22. We have 1 week to run it on private test set and get back to you if your docker have issue.
If you are afraid that your docker might not work with our testing pipeline and find the timeline too tight in case there are errors in your docker.
You can submit your model early. I will run the model every week and get back to participant with problematic submission.
You can always email if you have a specific issue about the submission. The final ranking will be announced after the end of the competition.
Thank you for your reply!
Posted by: darkness @ Nov. 23, 2022, 2:18 a.m.The final rank will be by SROCC only. PLCC, RMSE and runtime will only be used for reference.
Posted by: yixchen @ Nov. 29, 2022, 6:51 p.m.Another Question:
When I check README.md in the official GitHub Repo, it tells us that our submission should be less than 1G. Since our competition has two tracks (i.e., FR and NR), where does this limit take effect, each track or both tracks? In other words, is this 1G limit independent in each track or shared by both tracks?
You can create to two separate submissions for FR and NR if you wish to participate both. 1GB limit apply to one submission.
Posted by: ericalingyuan @ Dec. 7, 2022, 2:47 p.m.