1. Under the "Files" tab on the "Participate" page, click the download button for "#1 Development" to download the training data (with reference responses) and held-out development data (without reference responses).
2. Develop your system.
3. Evaluate your system's output on the development set.
- Generate the missing responses on the held-out development data.
- Include these results in a file named "answer.jsonl" following the expected submission format (https://sig-edu.org/sharedtask/2023#submission).
- Compress the "answer.jsonl" file with ZIP. For example, you could name this ZIP-compressed file "dev_submission.zip".
- Under the "Submit / View Results" tab on the "Participate" page, click on "Development".
- Fill in some metadata to describe your submission.
- Click on "Submit" to upload the ZIP-compressed file (e.g., "dev_submission.zip").
Once the results are submitted, the scoring program will run the evaluation metrics. If the submission succeeds, your results will be shown on the leaderboard.
Posted by: anaistack @ April 26, 2023, 7:50 a.m.