Hi:
I found a problem when I uploaded the result. And the leaderboard only showed the last commit. If so , I could not know which of the commits is the best. And the the LPIPS and FID evaluation score is not provided, so I can not know the difference within the others. So if convenient, could you share the LPIPS and PID score?
You have access to all your submissions, and you can check the performance on each one. I am not entirely sure I understand your issue. Can you please elaborate a bit more?
Regarding LPIPS and FID, that is the ideal case. Unfortunately, LPIPS and FID require a GPU in the codalab platform, which complicates things. I will take a look at that.
Posted by: afromero @ Feb. 23, 2022, 1:53 p.m.I understand your issue with the leaderboard. I noticed that a few days ago. I started working on a solution because the leaderboard is ranking the top three by using a different metric than the PSNR. Apparently, this is a known strange behavior in the platform https://github.com/codalab/codalab-competitions/issues/3017, and I am still trying to figure out a solution. Apologies for the inconvenience.
Posted by: afromero @ Feb. 23, 2022, 2:28 p.m.So what is the metric for the ranking top3 in the leaderboard if it is not the PSNR? If convenient, could you share with me?
Posted by: runzeer @ Feb. 24, 2022, 1:17 a.m.In this codalab version we put 5 different variables: PSNR, SSIM, Runtime per image [s], CPU [1]/ GPU [0], and Extra Data [1] / No Extra Data [0]. For some reason the bug I mentioned ranks the top 3 based on the last one (extra data). We are aware this makes no sense at all, and we are in communication with the codalab platform to fix this, if possible.
Posted by: afromero @ Feb. 24, 2022, 3:01 p.m.