The evaluation page states the following:
"There are 12 different environments present in the original dataset, from which three are available in the training data. A single (validation) environment among the 10 never-seen environments will be used to compute the OOD performance in "Development" phase, allowing challengers to select their preferred models for final phase. Another single (test) environment will be used to compute the OOD performance in the "Final" phase (see Generalization prize below). "
I'm a bit confused since 3 training + 10 never-seen environments add up to a total of 13 environments. Do I understand correctly that "Environment 3" in the training data is the same environment as is used to compute the OOD performance in the Development phase?
Posted by: bbussmann @ May 3, 2022, 1:43 p.m.Sorry this is a typo - it should be read "9" remaining environments for testing.
And no the test environments in dev and final phases are not part of training data.
HTH
Posted by: eustache @ May 4, 2022, 9:10 a.m.