The Autocast Competition - Forecasting the Future with AI Forum

Go back to competition Back to thread list Post in this thread

> Forecasting

Thank you for your feedback.

> It appears many of the questions are a 'historical fact lookup' when certain events have not occurred by 1-2 days prior
This is an important point. While distance to the closing date is a useful feature for some T/F questions, for the majority of questions it is not decisive. This is because questions can resolve unexpectedly (e.g., https://www.metaculus.com/questions/3474/when-will-queen-elizabeth-ii-cease-to-be-queen-of-the-united-kingdom/) and may have uncertainty until the closing date (e.g., https://www.metaculus.com/questions/11904/labor-merger-in-2022-israel-election/). To get a sense for how common these cases are, the Metaculus question search is a good resource: https://www.metaculus.com/questions/?order_by=-activity&main-feed&status=resolved

For questions that are not T/F questions, distance to the closing date is not itself an informative feature, but nowcasting/reading comprehension could still allow one to be very close to the correct answer (e.g., https://www.metaculus.com/questions/7971/ethereum-consensus-mechanism/). While this does make some questions easier, many questions still have uncertainty up to the closing date for the same reasons mentioned above, and reading comprehension is an important skill for all questions.

> is there any consideration of evaluating participants on the entirety of the time between 5/11/2021 and market close
We are considering this for future rounds.

> possible to achieve a very high score by leaning 'yes X happened' on dates long before those listed in the question
We agree that this observation provides some signal for T/F questions, and good models should take it into consideration, but it is probably not enough on its own to obtain a high score (see above response).

> a random sample of dates would match-up quite well with real-world forecasting ability.
This is a good suggestion and it would make the competition much more challenging. We'll focus on the simpler case of producing a final forecast in this round and may consider a similar setup in future rounds (see above response).

Posted by: Autocast @ Sept. 22, 2022, 3:29 a.m.
Post in this thread