Hello,
The ranking in the leaderboard seems to depend only on the number of parameters,
but according to the description the final evaluation will be done mainly on the compromise between the number of operations, the execution time and the fidelity,
and is not about the number of parameters.
I have a hard time understanding what is more important.
For example, for you what is the best between these two entries?
Entry 1: PSNR: 35.5 muPSNR: 35.5 time: 0.05s GMAccs: 130 num. of params: 65000
Entry 2: PSNR: 37.5 muPSNR: 36.1 time: 0.06s GMAccs: 195 num. of params: 3500000
Thank you
Posted by: steventel @ Feb. 15, 2022, 9:29 p.m.Hello steventel,
Please refer to the "Evaluation" tab in the codalab platform ( https://codalab.lisn.upsaclay.fr/competitions/1514#learn_the_details-evaluation ) to know the exact details about the evaluation criteria for this challenge Track.
In summary, there are two aspects to consider for this track: (1) eligibility and (2) evaluation.
For (1) eligibility, methods submitted to this track are required to operate in less than 200 GMAC operations, and runtime will also be considered specially in cases where number of operations and runtime diverge. This is just a requirement to participate in this track. The number of parameters play no role whatsoever in eligibility.
For (2) evaluation, methods will be evaluated and ranked based on the two PSNR metrics. There is plenty of information about how these two metrics are computed and evaluated in the "Evaluation" tab.
Complexity metrics will in principle only affect eligibility in this track, and ranking mainly comes from the fidelity scores. For methods where PSNR metrics are very close, complexity metrics might be used to disambiguate ranking.
For the example you wrote, both entries are eligible as they are both under 200GMACS. From both entries, "Entry 2" has higher PSNR scores so it will be ranked higher.
Best regards,
Edu.