Hi, forgive me if this information is posted somewhere but does the "OrganizerBaseline" score of 40.47 reflect mapping to AlexNet as shown in the example CoLab notebook? If not, how is this OrganizerBaseline computed so I can get a better understanding of it as a reference when submitting my own scores?
Thanks greatly,
Paul
Hi Paul,
For the "OrganizerBaseline" we extracted the feature maps from all AlexNet layers, appended them to each other (across layers), and reduced them to 100 PCA components (whereas in the tutorial the feature maps are instead extracted for a single chosen DNN layer, due to limited computational resources provided by Colab). Then, the linear regression training is just like in the tutorial, with the exception that with the "OrganizerBaseline" we trained the regression weights using all the available training data.
The Algonauts Team
Posted by: giffordale95 @ June 7, 2023, 5:45 p.m.Thank you!
Posted by: PaulScotti @ June 7, 2023, 6:13 p.m.Thank you for the info! And what is the Pearson correlation of this baseline (not noise-normalised)?
Posted by: aborovykh @ June 11, 2023, 7:14 a.m.The "OrganizerBaseline" corresponds to the noise-normalized encoding accuracy as described here (https://codalab.lisn.upsaclay.fr/competitions/9304#learn_the_details-evaluation). In the Colab tutorial, however, the the models are evaluated by simply correlating predicted and real data (without noise-normalization).
The Algonauts Team
Posted by: giffordale95 @ June 11, 2023, 7:20 a.m.Thanks - I get that. I was wondering what the value of the non-normalised correlation was. The reason I ask is I have a big discrepancy between Pearson correlation, my own noisenormalised value and the leaderboard one, so trying to find my mistake by replicating your solution. Thanks!
Posted by: aborovykh @ June 11, 2023, 7:31 a.m.Thanks for clarifying! The median (across subjects/hemispheres/vertices) of the correlation values between the ("OrganizerBaseline") predicted data and the test data is "0.32".
The Algonauts Team
Posted by: giffordale95 @ June 13, 2023, 7:58 a.m.