Competition

Evaluation

• Subtask 1: Binary text classification. The performance in this subtask is evaluated using the Precision, Recall and F1 scores. The "yes" label is treated as the positive class.
• Subtask 2A: Multi-label text classification. The micro-average Precision, Recall and F1 scores serve as the evaluation metrics. The "yes" label is considered the positive class for each item in the Insomnia rules (Definition 1, Definition 2, Rule A, Rule B, and Rule C).
• Subtask 2B: Evidence-Based Classification. The alignment of text spans provided by participants with the reference spans from the clinical notes is assessed using macro-average ROUGE-L Precision, Recall and F1 scores.

The evaluation scripts are available in the evaluation folder within the GitHub repository for the task.

Terms and Conditions

By submitting results to this competition, you consent to the public release of your scores at the SMM4H'25 workshop and in the associated proceedings, at the task organizers' discretion. Scores may include, but are not limited to, automatic and manual quantitative judgements, qualitative judgements, and such other metrics as the task organizers see fit. You accept that the ultimate decision of metric choice and score value is that of the task organizers. You further agree that the task organizers are under no obligation to release scores and that scores may be withheld if it is the task organizers' judgement that the submission was incomplete, erroneous, deceptive, or violated the letter or spirit of the competition's rules. Inclusion of a submission's scores is not an endorsement of a team or individual's submission, system, or science. You further agree that your system may be named according to the team name provided at the time of submission, or to a suitable shorthand as determined by the task organizers. You further agree to submit and present a short paper describing your system during the workshop. You agree not to redistribute the training and test data without the explicit approval of the organizers.

Contact Information

Guillermo Lopez-Garcia, Cedars-Sinai Medical Center, USA (Guillermo.LopezGarcia@cshs.org)

Submission Format

For each subtask, ground truth annotations are provided in JSON format. Participants are required to submit their system outputs following the same format as the ground truth annotations provided by the organizers.

• Subtask 1: Binary text classification. System predictions should be submitted as a ZIP file containing a single JSON file named "subtask_1.json". Participants must predict the Insomnia status of each note by providing a "yes" or "no" label. Please see this sample JSON file for guidance on how to format the predictions for Subtask 1.
• Subtask 2A: Multi-label text classification. System predictions should be submitted as a ZIP file containing a single JSON file named "subtask_2a.json". Participants must predict a "yes" or "no" label for each item of the Insomnia rules: Definition 1, Definition 2, Rule A, Rule B, and Rule C. Please see this sample JSON file for guidance on how to format the predictions for Subtask 2A.
• Subtask 2B: Evidence-Based Classification. System predictions should be submitted as a ZIP file containing a single JSON file named "subtask_2b.json". For items Definition 1, Definition 2, Rule B, and Rule C, participants must provide a label ("yes" or "no") and include a list of specific text spans from the note that justify each classification. Note that text spans are required only when the corresponding item is assigned a "yes" label. For items assigned a "no" label, participants should submit an empty list [] for the text spans, indicating no justification is required. Please see this sample JSON file for guidance on how to format the predictions for Subtask 2B.