On the 18th, when this secondary competition was open, I briefly saw a submission from user chowdhury32 with the team name "TEST LABELS COMPROMISED" and an f1 score of 1. More recently, there's been a submission from user batohoy with a perfect f1 score as well.
Posted by: Lancelot53 @ Aug. 27, 2023, 7:28 a.m.I just found out a file `csv_dir/test_set_labels.csv` in the official repository VIPCUP2023_OLIVES at one of the commits. Just type `git checkout 28dde3d` inside repo and the file will appear.
Posted by: batohoy @ Aug. 27, 2023, 7:39 a.m.maybe you forked it beforehand, but the current repo (https://github.com/olivesgatech/VIPCUP2023_OLIVES) doesn't have that commit.
Posted by: Lancelot53 @ Aug. 27, 2023, 7:45 a.m.I cloned the repo to have local copy. You are right, it seems there is no such commit at official repository (https://github.com/olivesgatech/VIPCUP2023_OLIVES). I guess somebody must deleted it.
I will create a public repository and share it.
but if you search for VIPCUP2023 at github, you can still find a fork of old repository and it looks like it has 9 forks at github too. Here is the link to the commit I was talking about:
https://github.com/pranavmkoundinya/VIPCUP2023_OLIVES/tree/28dde3dee13dccfd1e6bf7239aa71d2aa14c1c7a
and the test_set_labels
https://github.com/pranavmkoundinya/VIPCUP2023_OLIVES/blob/28dde3dee13dccfd1e6bf7239aa71d2aa14c1c7a/csv_dir/test_set_labels.csv
Thanks for being honest and up front about it, not everyone would take that road. I guess the test set labels were indeed leaked. Months of work... Poof.
Posted by: Lancelot53 @ Aug. 27, 2023, 7:56 a.m.Gee, at first when I found it I wondered how it could be used to improve model, now that I think about it this competition is indeed already pointless. I wonder how many of us, the participants of the cup, planned to use those labels somehow to win.
Posted by: batohoy @ Aug. 27, 2023, 8:15 a.m.but the people organising the competition must have known about it, since they removed the original repository, and as you mentioned, there was someone who also received a 1.0 score with the team name "TEST LABELS COMPROMISED".
Posted by: batohoy @ Aug. 27, 2023, 8:18 a.m.Hello,
Yes, we were very recently made aware of the label leak in the github repository and are taking taking this seriously. We have taken steps to address this and we will have a full and transparent forum post tomorrow at the end of Phase 1.
To ensure a fair competition:
1. We are providing a new testset for personalization in phase 2. The winners of the competition will be judged on this new testset. None of the data or patients are part of the current testset on codalab. Having a generalizable model on the current phase will help you in the new testset. Note that, as has always been the plan, phase 2 will occur offline and participants will be asked to submit final results through CMS.
2. The new testset for phase 2 will not be derived from a clinical trial, so it is in the participant's best interests to not overfit their models or directly use the test labels in codalab validation.
3. Since the current labels are compromised, we will allow all teams to go through to Phase 2.
We will have a full forum post and email tomorrow regarding the leak and the new testset. We apologize for any inconvenience and assure everyone that we are working diligently to run the competition smoothly and fairly.
Posted by: OLIVES @ Aug. 27, 2023, 9:40 a.m.Do we need to submit notebook and document ,as we are supposed to do so . Is it necessary for entering phase II?????
Posted by: Rogers @ Aug. 27, 2023, 12:44 p.m.No it is not necessary for entering Phase 2. We will ask for them at the end of Phase 2, to be submitted through CMS website where you have registered.
Posted by: OLIVES @ Aug. 27, 2023, 1:14 p.m.