I noticed that the test data comes from DIV2K , and the label(or the 4 times downsampled data) can be found easily online. To ensure the fairness of the challenge, is there some supplement test data to check whether the model is overfitting in test data?
Posted by: jetblock @ July 26, 2022, 2:59 a.m.yes, there is; the reproducibility of the results is checked
Posted by: Radu.Timofte @ July 26, 2022, 7:34 a.m.Hi, I just have one similar question: how to check if a model is trained with the validation data provided in the development phase ? Unlike the test data, the ground truth of DIV2K validation data is public and released on the official website.
Posted by: mingxi @ July 26, 2022, 8:57 a.m.It is hard to control if someone trains their model with the validation data. However, for the test phase, all participants are requested to submit their pre-trained model before we release the test data, it is able to avoid fine-tuning the models on the test set or the data which are similar to the test data.
Posted by: RenYang @ July 26, 2022, 5:20 p.m.I understand the concern about prevention of overfitting the test set by submit code before test data release, but as the test data are publicly available (DIV2K test input with official degradation code), I think this code/model pre-submission can hardly take effect if someone want to overfit the test set. If someone did the overfitting before test set release by using the public test input for finetuning, how can this be detected?
Posted by: jzsherlock @ July 26, 2022, 5:34 p.m.As far as I know, the test data (including the low-resolution data) are not publicly released, refer to https://data.vision.ee.ethz.ch/cvl/DIV2K/. Where did you find the test data publicly?
Posted by: RenYang @ July 26, 2022, 5:42 p.m.Sorry for the wrong information. I checked again and found that I confused the DIV2K dataset with another dataset REDS, which used in another AIM track that I participate. The REDS test input is publicly available, and DIV2K has no public test input. Therefore the pre-submission do have benefit for preventing overfitting and keep the fairness of the challenge.
Posted by: jzsherlock @ July 26, 2022, 7:35 p.m.Hi,
Thanks for your reply ! I agree that it is really hard to control and check.
However, even if a model was not fine-tuned directly on the test data, but trained with the DIV2K validation data, it would still enjoy more performance improvement than the same model trained with other extra datasets, because the val data and test data are from the same source dataset and do not suffer severe domain discrepancy. So it still seems unfair to compare it with other methods.
So my suggestion, is it possible to report model performance on both the val data and test data ? I think if a model was trained with the val data, it would suffer much more performance degradation on the test data compared with models not. Does it work in the correct way ?
Posted by: mingxi @ July 27, 2022, 12:14 a.m.reply for @RenYang,"As far as I know, the test data (including the low-resolution data) are not publicly released, refer to https://data.vision.ee.ethz.ch/cvl/DIV2K/. Where did you find the test data publicly?"
low-resolution test data is easy to be found online, for example, https://github.com/ofsoundof/IMDN/tree/main/data/DIV2K_test_LR
1. We cannot control the training data, and extra trainning data are allowed to be used in the Challenge. When checking the reproducibility, we will also check the generalization, e.g., if a method only works well on a specific dataset.
2. It might be possible to find the low resolution test data in previous Challenges. We just follow the previous Challenges to use DIV2K as the test set. What we can do to prevent overfitting is to check the generalization of the submitted models, e.g., if a model only works well specifically on the test set
Posted by: RenYang @ July 27, 2022, 4:31 a.m.1. We cannot control the training data, and extra trainning data are allowed to be used in the Challenge. When checking the reproducibility, we will also check the generalization, e.g., if a method only works well on a specific dataset.
2. It might be possible to find the low resolution test data in previous Challenges. We just follow the previous Challenges to use DIV2K as the test set. What we can do to prevent overfitting is to check the generalization of the submitted models, e.g., if a model only works well specifically on the test set
Posted by: RenYang @ July 27, 2022, 4:31 a.m.1. We cannot control the training data, and extra trainning data are allowed to be used in the Challenge. When checking the reproducibility, we will also check the generalization, e.g., if a method only works well on a specific dataset.
2. It might be possible to find the low resolution test data in previous Challenges. We just follow the previous Challenges to use DIV2K as the test set. What we can do to prevent overfitting is to check the generalization of the submitted models, e.g., if a model only works well specifically on the test set
Posted by: RenYang @ July 27, 2022, 4:31 a.m.There is a suggestion. If someone cheat in this competition (for example, use validation or test data for training), and be found by your generalization test, you should public their real name and institution，it will be a threaten. None would take the chance. : )
Posted by: chenmigongzuo @ July 28, 2022, 9:53 a.m.Maybe some unknown dataset can be used by organizers as a private dataset to supplement the test performance.
Posted by: deepernewbie @ July 28, 2022, 3:51 p.m.