dear organizers,
this track is only evaluated using the Places dataset (info at the website). Therefore, we must submit only 7 x 1000 test images corresponding to the Places test images, is it right? can you provide a sample submission? Since the number of submissions in the test phase is limited I want to make sure about it :)
"""
Test Data for the 4 datasets: https://polybox.ethz.ch/index.php/s/qnf3NglUDqmvGBr.
It contains 1,000 images for each type of mask (7 x 1,000), for each dataset.
"""
Hello, this track is evaluated on Places and on FFHQ. The folder structure should be the same as for the validation one. There is a sample here: https://codalab.lisn.upsaclay.fr/my/datasets/download/6be1f644-0eee-49a5-9e4f-ddee7152c1ad
That link contains samples for the validation codalab server, that is, only certain frames and for the Places dataset. Your folder structure should be the same, yet with both Places and FFHQ, and all the frames. We expect both validation and test solutions.
Keep in mind the additional information regarding the factsheet.
Please let us know whether this information was clear.
Posted by: afromero @ March 24, 2022, 12:47 p.m.Hello, thanks for pointing out the demo submission.
I think then, the description at the website is wrong, it says the evaluation is ONLY for Places dataset.
"""Track 2 is only evaluated over the Places dataset, by using automated Semantic Segmentation annotations. See code."""
"""Therefore, in the validation phase, we will invite the participants to submit their results with *ONLY* the Places dataset, and in a uniform sampling each 10th frame. This means, images 000000.png, 000010.png, 000020.png, 000030.png, 000040.png, … for all the masks."""
I understand that the structure will be as follows, using ONLY the test (Test Data for the 4 datasets: https://polybox.ethz.ch/index.php/s/qnf3NglUDqmvGBr), without sampling each 10th frame:
FFHQ/ #7000 images
Places/ #7000 images
readme.txt
if possible I would like to ask the host to increase the number of submissions during the test phase, as debugging this will probably take a couple of submissions.
Thank you.
Hello,
You are right, we completely missed that part. Apologies for the inconvenience. We highlighted in the "Evaluation" section as "At test time, this track is evaluated over those datasets with semantic labels, that is, FFHQ and Places.", but forgot to highlighted in the "get data" section. In any case, this should not be a problem as you probably tuned your parameters for Places, and FFHQ should not require expensive additional tuning. Considering that there is no FFHQ server feedback, can you please elaborate on how getting an additional number of submissions would help? During the test phase, there is no feedback, as we will manually download all your data (from google drive, or your own server, etc) and manually compute the FID, LPIPS, PSNR, etc.
Regarding the structure, you are right. It would be for all the frames in the validation set and all the frames in the test set. Additionally, you should include the factsheet. Keep in mind that the submission on the Codalab server is restricted in size, and all the data would be several GB, and that is why we require all the frames as a shared link.
Please, let us know if we addressed correctly your questions.
Posted by: afromero @ March 24, 2022, 2:09 p.m.Following your concerns, I just sent an email that hopefully clarifies this a bit more.
Please let us know if it was clear. I will put it here just in case:
"Codalab failed to upload file":
Just to clarify some information, the Codalab submission is restricted in size, and all your generated frames are naturally several GBs, which will produce an error in the system. In order to avoid this and to get a successful submission, you can follow the same procedure as in the validation stage (every 10th frame), and include the link to ALL your frames (validation and test) in the README in the "Other description" part.
"Test Feedback":
During the test phase, the feedback you get from Codalab is using the same Ground-Truth as in the validation phase, so those metrics do not correspond to the real ones. For the test stage, we will manually download your data and compute the corresponding metrics locally.
hello,
In final submission track 2, we just send a e-mail to you? or we also should submit to codalab?
Hello,
If you send us an email, please ask us for a confirmation reply just to be on the safe side. In that case, it is fine if you do not submit it to codadlab.
I am having problems to upload the submission, upload time limit I think (not sure if it's my connection or codalab server).
Is there any problem if I provide the information directly ok the report with the download links (as a kaggle dataset, easier to download than Google drive) for full validation and test?
I will email all the details.
Thank you.
Posted by: nanashi @ March 29, 2022, 11:33 p.m.Hello, yes it’s fine.
Posted by: afromero @ March 30, 2022, 12:35 a.m.