hi,
I set the out = torch.nn.functional.interpolate(input, scale_factor=2,mode='bicubic') and get 31.485 PSNR with 0.14ms per image(1020x540 to 2040x1080) on my RTX 3090. According to the scoring function from https://codalab.lisn.upsaclay.fr/competitions/10227#learn_the_details-evaluation:
1. the score of bicubic is 214
2. a cnn model run 1000 fps and 34.3 PSNR, which score is 211 and slightly lower than bicubic
But no one can reach 34.3 PSNR on the leaderboard now. Bicubic is the best unless someone's model get 1000+ fps, which is unreasonable. I think a PSNR baseline is needed, for example, PSNR should higher than 32.
By the way, does the final testset also contain DIV2K testset like the valid set? Thank you~
I think a good score function should satisfy the following conditions:
1, when running time is far less than 16ms, PSNR dominates the score function ;
2, when running time is far higher than 16ms, running time dominates the score function ;
3, when running time is around 16ms, the score function connect 1&2 smoothly;
4, when the PSNR is no higher than that of bilinear/bicubic/lanczos upsampling, the score should be around 0.
5, when the running time is higher than that of RFDN/IMDN, the score should be around 0.
Hello, thanks for your feedback,
- The HR test images are not DIV2K, are native 4K and 8K (cropped)
- The final output of the model should be always 4K 3840 x 2160
- The scoring function is based on our previous competitions "Efficient and Accurate Quantized Image Super-Resolution on Mobile NPUs, Mobile AI & AIM 2022 challenge: Report" Section 2.5 (https://arxiv.org/pdf/2211.05910.pdf)
We appreciate your feedback and therefore we will study a new final scoring function and notify everyone
- Organizers.
Posted by: nanashi @ Feb. 22, 2023, 10:45 a.m.Hi~ thank you for giving consideration to the problem of the calculation formula.
We hope a new scoring formula can be made available as soon as possible. Appreciate it.
Hello,
we updated the evaluation page: https://codalab.lisn.upsaclay.fr/competitions/10228#learn_the_details-evaluation and the repo https://github.com/eduardzamfir/NTIRE23-RTSR/blob/master/demo/calc_scoring.ipynb
The formula is derived from other challenges like "Efficient and Accurate Quantized Image Super-Resolution on Mobile NPUs, Mobile AI & AIM 2022 challenge: Report" (https://arxiv.org/abs/2211.05910).
Having a piece-wise function is tricky, we decided to penalize the submissions bellow Interp. Methods (Bicubic) such that score is 0 (null) for those methods and worst than those. Usually in the literature, Bicubic runtime is ignored.
Improvements on PSNR and runtime are more consistent now (+0.1dB gives the same boost as -1ms approx).
You can include other potential formulas in your challenge report submission, and we will consider adding them in the final report. However, we will use for scoring https://codalab.lisn.upsaclay.fr/competitions/10228#learn_the_details-evaluation
Methods like IMDN and RFDN are not competitive in this setup, we proposed a baseline for 30FPS https://github.com/eduardzamfir/NTIRE23-RTSR#performance-of-baseline-methods
In general, the faster the better. Methods above 42ms7img (24FPS) will not be considered acceptable in this setup.
Thanks for your feedback.
- Organizers.