Hello,
In light of a recent loophole in the detection track validation networks found by one of the participants, we are appending a section to the rules that disallows solutions of this type. Note that this loophole only affects the detection track, but the new rules apply to all tracks. The addition to the rules is as follows:
"
The use of features that are clearly loopholes is not allowed (e.g., differences in Python classes, batch norm num_batches_tracked statistics, or other discrepancies that would be easy to patch but may have been overlooked). As this is a new format of competition, we may not anticipate all loopholes and we encourage participants to alert us to their existence. Legitimate features that do not constitute loopholes include all features derived from the trained parameters of the networks.
"
We will release a new version of the validation set for the detection track that fixes this loophole by Sunday (7/23). Participants in the detection track will need to redownload the validation set (a new Zenodo link will be sent). All submissions on the old validation set will be removed from the leaderboards at that time, but *in the meantime we will continue to accept submissions on the existing validation set*. Results on the new validation set may differ by a small amount on the new validation set due to different initializations and random seeds.
All the best,
Mantas (TDC co-organizer)
Hello,
The new version of the detection track validation set is available here along with the other datasets: https://zenodo.org/record/6894041. This is intended as a drop-in replacement for the old tdc_datasets folder. All competition materials (dataset links on the website and CodaLab pages, and the download script in the starter kit) have been updated accordingly. Alternatively, if you only want to download the new validation set without redownloading the rest of the datasets, please see this separate upload: https://zenodo.org/record/6894041 and replace the old validation set with the new one.
The validation leaderboard has been reset and all previous submissions have been deleted. You can recompute scores for the new validation set using the exact same code as before. Note that the new validation set is more challenging than the old one, with the AUROC for the MNTD baseline being 9 points lower. After running various tests, we suspect that this is due to random variation in the attack specifications, which are sampled from a wide distribution. Importantly, this should not significantly affect the relative ordering of methods.
All the best,
Mantas (TDC co-organizer)