Dear Organizer,
as described in https://cvnlab.slite.page/p/AGEte5w9Nq/Overview-of-the-data
"the behavioral data and experimental design information (including the actual stimuli shown) for the held-out scan sessions are still available."
as described in challenge rule:
"Participants can use any (external) data for model building and any model. However, participants may not use actual brain responses measured for the images in the test set (from any source). Also, if you ever had access to the fMRI test data you cannot participate in the challenge."
I want to make sure, is it OK to use behavioral data and experimental design, for this challenge. These two open up whole new possibilities.
Best regards, huze
Posted by: huze @ July 13, 2023, 9:16 a.m.we can found the data in:
https://cvnlab.slite.page/p/fRv4lz5V2F/Behavioral-data
https://cvnlab.slite.page/p/NKalgWd__F/Experiments
Dear huze,
Thanks for pointing out this issue! Yes, it is allowed to use behavioral data relative to the challenge TRAIN split to train the encoding models. However, using the behavioral data relative the challenge TEST split defeats the spirit of the challenge: one should not use any information related to the challenge test split to train the models (this also includes the experimental design). We will require the challenge winners to submit a short report of their models, along with the corresponding code: this will mitigate the risk of people cheating (i.e., of people using any information relative to the test split to train their models).
The Algonauts Team
Posted by: giffordale95 @ July 14, 2023, 10:14 a.m.Dear Organizers,
About the statement "one should not use any information related to the challenge test split to *train* the models (this also includes the experimental design).": Clearly experimental design for the test data should not be used for training the models, but can it be used for the predictions (without use during training)? Could you clarify if experimental design for the test data should not be used at all, and, if so, whether they were shared by mistake?
Best regards,
ashes
Dear ashes,
More specific answer. What exactly are you referring to with "experimental design"? Is it the order with which the images were presented? Is it the scanning sessions on which the images were presented? It it the task? Is it other?
More general answer. It is good to always keep in mind what is the ultimate goal of the Challenge: fostering the development of computational models of the brain that will hopefully benefit both computational neuroscientists and computer scientists. Now, if these models end up being good out of some "tricks" such as overfitting to the test split, or using behavioral responses from the test split, or leveraging experimental design choices idiosyncratic to NSD, or other similar things ... how will this result in models beneficial to other scientists? In other words, the spirit of the challenge is not only to come up with model which best predict brain responses, but rather to come up with models which do so while at the same time being useful (e.g., Would these models also perform well on other datasets? Could these models give any new insights on biological and artificial intelligence?).
The Algonauts Team
Posted by: giffordale95 @ July 17, 2023, 8:42 p.m.