Competition

Evaluation Criteria

The results will be evaluated with 1) Exact Match score, and 2) F1 score.

Prepare your prediction file into the following format, as a list of dictionaries, each dictionary contains three fields: 1) "uid": the uid of QA example, 2) "predicted_program": the predicted program, 3) "predicted_ans": the predicted_ans. The predicted program is a list of predicted program tokens with the 'EOF' as the last token. For more details, please refer to the Github.

Please zip the generated test prediction file named as test_predictions.json into test_predictions.zip

Terms and Conditions

It is expected that academic integrity must be strictly followed. You should submit original, real test results of your system. No cheating allowed.