Hi, can you please provide additional information about the evaluation metric? regardless the confidence-based adjustments it is counter-intuitive that lower overall RMSE results in worse RMSE_MAPPED performance.
Posted by: amsqr @ June 14, 2022, 8:31 p.m.hey,
Please note that the bug in validation set is fixed and you should submit your predictions again to see its actual statistics.
About RMSE_MAP:
What we do is to fit a linear function between predicted_mos (submitted by a team) and the true MOS from subjective results.
Then, we apply the function on the predicted_mos (i.e. mapping them) to created predicted_mos_mapped. As a result we remove the offset and gradient between predicted and the true MOS.
Finally we calculated the RMSE between the "predicted_mos_mapped" and "true MOS".
I hope it helps. For further information about the offset removal you may check the ITU-T Rec. P.1401 section 7.3.
Posted by: qu.lab @ June 15, 2022, 2:13 p.m.