The competition is completed in two phases (phase1, phase2). In order to successfully complete the evaluation results, participants need to pay attention to the following points:
Phase1: train_label.txt is used to train the model (Label: live=0, fake=1). Then the trained model is used to predict the sample scores in dev.txt. Participants can directly submit the predicted score file in codalab system. In order to submit results at one time, participants need to combine predicted files from 3 protocols together.
(1) merge order: dev.txt(p1), dev.txt(p2.1), dev.txt(p2.2)
(2) merge method: Continue straight by column.
Note that the order of the samples in dev.txt cannot be changed. The final submitted file contains a total of 15,206 lines. Each line in the file contains two parts separated by a space. The first part is the relative path of each image, and the second part is the prediction score given by the model (representing the probability that the sample belongs to the live face). Such as:
dev/000001.jpg 0.15361 #Note: line 1- the first row of dev.txt of p1
......
dev/006000.jpg 0.23394 #Note: line 6,000- the last row of dev.txt of p1
dev/000001.jpg 0.15361 #Note: line 6,001- the first row of dev.txt of p2.1
......
dev/005006.jpg 0.23394 #Note: line 11,006- the last row of dev.txt of p2.1
dev/000001.jpg 0.15361 #Note: line 11,007- the first row of dev.txt of p2.2
......
dev/004200.jpg 0.23394 #Note: line 15,206- the last row of dev.txt of p2.2
Finally, the participants should compress the predicted file into a ZIP file (do not add any folder in the ZIP) for final submission. The ZIP file can be named randomly (such as your team-name). We will feedback on the testing results online, including the results of 3 protocols seperately. In order to fairly compare the performance of participants' algorithms, this competition does not allow the use of other training datasets and pre-trained models.
Phase2: After phase1, we will release the testing data test.txt in phase2, plus the label of development data dev_label.txt (Label: live=0, fake=1) which is used by the contestants to select the best model. In order to submit results at one time, participants need to combine the 2 predicted files (dev.txt, test.txt) in the same protocol into one file before result submission via codalab system. Also, participants need to merge the combined txt file of each protocol into a new final txt file. Note that the order of the samples in test.txt cannot be changed.
(1) merge order: dev.txt(p1)，test.txt (p1), dev.txt(p2.1)，test.txt (p2.1), dev.txt(p2.2)，test.txt (p2.2)
(2) merge method: Continue straight by column.
The final merged file (for submission) contains a total of 72,618 lines. Each line in the file contains two parts separated by a space. Such as:
dev/0000/000001.jpg 0.15361 #Note: line 1- the first row of dev.txt of p1
......
dev/0089/006000.jpg 0.23394 #Note: line 6,000- the last row of dev.txt of p1
test/0000/000001.jpg 0.15361 #Note: line 6,001- the first row of test.txt of p1
......
test/0164/021506.jpg 0.23394 #Note: line 27,506- the last row of test.txt of p1
dev/0000/000001.jpg 0.15361 #Note: line 27,507- the first row of dev.txt of p2.1
......
dev/0089/005006.jpg 0.23394 #Note: line 32,512- the last row of dev.txt of p2.1
dev/0000/000001.jpg 0.15361 #Note: line 32,513- the first row of test.txt of p2.1
......
dev/0089/009900.jpg 0.23394 #Note: line 42,412- the last row of test.txt of p2.1
dev/0000/000001.jpg 0.15361 #Note: line 42,413- the first row of dev.txt of 2.2
......
dev/0089/004200.jpg 0.23394 #Note: line 46,612- the last row of dev.txt of p2.2
dev/0000/000001.jpg 0.15361 #Note: line 46,613- the first row of test.txt of p2.2
......
dev/0089/026006.jpg 0.23394 #Note: line 72,618- the last row of test.txt of p2.2
Importantly, in the phase1, we will first find the best threshold on the prediction score of the development data (dev.txt), and then use the defined threshold to calculate the ACER on the development data (dev.txt). In the phase2, we will first find the best threshold on the prediction score of the development data (dev.txt), and then use the defined threshold to calculate the ACER on the test set (test.txt).
Posted by: AjianLiu @ Feb. 23, 2024, 6:42 a.m.