Hi participants,
We have an updated evaluation script, so if possible, please re-run your submissions. Our original script didn't check for duplicates, so if your HPO code outputted the same point that happens to be Pareto for each iteration, your score would be inflated. This is now fixed.
Best,
Kevin
Follow-up on this thread: the max number of Pareto points is 7 in set1 (so-en) and 14 in set2 (sw-en). If there are some old results with #pareto above those, they should be ignored.
To be clear about what is expected in an output 'so-en.predict’ file that is written out every time you run the ingestion program with your HPO algorithm, each line represents a full HPO run. In the following, a sample like 332 is sampled multiple times; this is not desired. The old evaluation script would double-count 332 as correct multiple times; the new one would not. In any case, it is in your best interest to not waste an iteration on previously-seen samples.
318 445 476 537 99 99 318 398 168 225 343 537 323 332 510 229 87 582 332 332 233 139 82 101 332 366 332 409 108 34 366 82 184 332 332 332 332 261 124 332 559 171 332 124 19 332 254 87 87 573 171 332 87 332 171 147 254 263 87 34 124 360 332 263 332 34 292 332 330 330 332 124 332 332 171 171 573 332 124 124 310 332 87 171 124 332 366 124 241 332 171 171 171 332 47 493 34 544 254 498 332 87 171 559 324 171 12 124 332 514 310 171 87 171 87 215 27 401 171 371 87 171 205 332 22 332 144 254 87 310 493 263 171 124 254 201 99 47 171 65 330 51 217 493 217 201 332 310 457 537 87 488 171 284 139 171 359 115 310 493 514 310 310 205 332 493 488 323 394 332 171 205 123 310 87 511 171 315 133 585 514 199 493 332 34 493 310 171 488 310 310 554 310 310 171 347 205 99 310 310
Posted by: kevinduh @ May 12, 2022, 1:09 p.m.