FYI, in some of the tests in scenarios the random algorithm did better than the baseline shortest path. Here are two examples:
Test18/Level0 (random: 2690.279 baseline: 2710.722)
Test18/Level1 (random: 2604.195 baseline: 2689.439)
This situation means that scores that are less than (better) than both will have a negative normalized score.
Here's a full list of tests in scenarios that will produce reversed normalized scores:
Test_15/Level_1.pkl
Test_16/Level_1.pkl
Test_16/Level_2.pkl
Test_16/Level_6.pkl
Test_17/Level_7.pkl
Test_17/Level_8.pkl
Test_18/Level_0.pkl
Test_18/Level_1.pkl
Test_18/Level_2.pkl
Test_18/Level_3.pkl
Test_18/Level_4.pkl
Test_18/Level_5.pkl
Test_18/Level_6.pkl
Test_18/Level_9.pkl
Test_19/Level_0.pkl
Test_19/Level_1.pkl
Test_19/Level_2.pkl
Test_19/Level_3.pkl
Test_19/Level_4.pkl
Test_19/Level_6.pkl
Test_19/Level_8.pkl
Test_19/Level_9.pkl
Thanks for bringing this issue up. We're looking into this.
Posted by: ccafeccafe @ Jan. 20, 2023, 4:13 p.m.jkolen, thanks so much for letting us know, and for taking the time to write out the details.
You are certainly right - we missed this issue. We fixed it in a couple of ways:
1) We used a different baseline (similar to the original, but with a few differences). We will share this baseline next week.
2) We were still seeing the issue in a couple of levels. The baseline solution will sit and wait for routes to become available, and it appears that in some cases a random movement pattern actually beats this. After some discussion, we decided to simply omit these levels.
We pushed an updated set of scenarios to https://airliftchallenge.com/scenarios/airlift_test_scenarios.zip (we also made changes to the parameters of the scenarios to improve scoring).
Posted by: abeckus @ Jan. 22, 2023, 1:23 a.m.