Hello,
The test phase for the primary round of the competition has started and will conclude one week from now. Please see the points below for important information regarding the test phase.
- Each team will be allowed 5 submissions total. CodaLab only has functionality for per-participant limits on the number of submissions, so we will enforce this by looking at the aggregate submissions across all members from a team and removing all submissions except for the first 5 successful submissions made by that team. To reduce the workload for organizers, we ask that team members coordinate with each other to avoid crossing the 5 submission limit.
- The leaderboard is private. The scores for submissions will not be visible on CodaLab (at least for the next week), and the best submission out of the 5 allotted to each team will determine that team's rank in the test leaderboard.
- The existing Zenodo pages for the individual tracks have been updated to include the test networks. These are available at the following links.
Detection: https://zenodo.org/record/7213750
Target Label Prediction: https://zenodo.org/record/7213751
Trigger Synthesis: https://zenodo.org/record/7213752
Evasive Trojans: https://zenodo.org/record/7213753
We will also release new Zenodo records containing only the test set networks within the next day.
- The validation phase for all tracks except the Evasive Trojans track will remain open during the test phase for teams that wish to continue developing their methods before submitting to the test set. (Note: the starter kit for the Evasive Trojans track contains code for self-evaluation, which teams may use to further develop their method during the test phase.)
All the best,
Mantas (TDC co-organizer)
Is the test phase closing now? Would it not be possible to keep it open until the rest of the competition?
Posted by: IRI @ Oct. 24, 2022, 7:48 a.m.Hello,
The purpose of the test phase was for participants to evaluate methods developed during the validation phase. If you did not have an opportunity to submit to the test phase, we may be able to extend it for a few hours to allow submission. However, out of fairness to the other participants we cannot extent it for other reasons (e.g., it would be unfair if we extended it to allow other teams to continue developing their methods).
All the best,
Mantas (TDC co-organizer)
That would be much appreciated.
Posted by: IRI @ Oct. 24, 2022, 3:40 p.m.OK. I will reopen the test phase for the next 6 hours.
Posted by: mmazeika @ Oct. 24, 2022, 5:27 p.m.Submissions to the detection track test phase have been reopened for the next 5.5 hours.
All the best,
Mantas (TDC co-organizer)
Hello,
Other participants have notified us that a grace period could introduce unfairness. To preserve fairness for other participants who submitted to the test phase during the past week, we have removed the grace period.
All the best,
Mantas (TDC co-organizer)