Just two questions on how the full-reference metrics will be tested:
* Should reference metrics predict MOS or DMOS (given that reference videos have different MOS values)?
* Will a non-linearity be fitted (mapping from metric predictions to MOS/DMOS values) before computing PLCC and RMSE?
Posted by: mantiuk @ Nov. 14, 2022, 1:41 p.m.1. Should reference metrics predict MOS or DMOS (given that reference videos have different MOS values)?
It should predict MOS but within the range of [0,100] as a float point number.
2. Will a non-linearity be fitted (mapping from metric predictions to MOS/DMOS values) before computing PLCC and RMSE?
Yes. Please check https://github.com/Easoncyx/2023_wacv_vqm_challenge for the non-linear transform used.
Thank you for the clarification.
My question regarding MOS vs. DMOS was ambiguous, so please let me clarify.
MOS values are sometimes corrected for the reference:
MOS_corrected = MOS_test - MOS_reference + 100
so that all reference images have the same score of 100. Such correction is done to eliminate potential differences in the quality of the reference content.
Should we assume that no such correction will be introduced?
Posted by: mantiuk @ Nov. 14, 2022, 10:07 p.m.Thank you for your question. No such correction is introduced when creating the dataset. You can find that the reference video MOS is not 100 in the training set, same goes for the private test set.
Posted by: yixchen @ Nov. 14, 2022, 10:37 p.m.I am still lost on the answer of first question here.
If I understood correctly, the metrics should predict MOS instead of DMOS for full-reference case since you already made the DMOS calculation and reported it as MOS in the training set.
where DMOS can be defined as suggested by mantiuk as DMOS = [MOS_test - MOS_reference + 100]
Hi concombr,
The training set has Sureal (https://github.com/Netflix/sureal) MOS recovered directly from the study. It is not DMOS. For example, you can find that reference videos (named 4k_ref_*) have scores less than 100.
Let me know if you still have questions.