Competition

Evaluation Metrics

• Subtask 3a: Given an observation, participants of this subtask will be required to submit the HPO term IDs of all key findings mentioned in the observation. This subtask will use the standard F1, Precision and Recall metrics, with the outcomes defined as follow:
  • True Positive: a mention of an HPO term in an observation labeled and normalized by an annotator, correctly detected and normalized by a system
  • True Negative: a mention of normal finding correctly ignored by a system - note that a normal finding can be an HPO term negated
  • False Positive: an HPO term incorrectly detected or normalized by a system - i.e. this HPO term was either not labeled by an annotator as being mentioned in the observation, or the term was mentioned but it was negated or normalized with a different HPO term ID
  • False Negative: a mention of an HPO term in an observation labeled and normalized by an annotator but not detected or incorrectly normalized by a system
• Subtask 3b: Given an observation, participants will be required to submit the spans of the key findings and their corresponding HPO term IDs. This subtask will use the standard metrics: Exact and Overlapping Precision, Recall and F1-score, with the outcomes defined as follow:
• Exact:
  • True Positive: a span, possibly disjoint, of a mention of an HPO term in an observation labeled and normalized by an annotator, correctly detected and normalized by a system, with the span predicted matching exactly the span labeled
  • True Negative: a mention of normal finding correctly ignored by a system
  • False Positive: a span incorrectly detected OR normalized by a system - i.e. the span predicted by the system was not labeled by an annotator as mentioning an HPO term, or the span predicted was a mention of a term but it was negated, or the predicted span and the labeled span differ, or the spans are identical but the span predicted is normalized with an incorrect HPO term ID
  • False Negative: a span of a mention of an HPO term labeled and normalized by an annotator but not detected OR incorrectly normalized by a system
• Overlapping:
  • True Positive: a span, possibly disjoint, of a mention of an HPO term in an observation labeled and normalized by an annotator, correctly detected and normalized by a system, with the span predicted overlapping exactly or partially with the span labeled
  • True Negative: a mention of normal finding correctly ignored by a system
  • False Positive: a span incorrectly detected OR normalized by a system - i.e. the span predicted by the system was not labeled by an annotator as mentioning an HPO term, or the span predicted was a mention of a term but it was negated, or the spans predicted and labeled differ, or they overlap but the span predicted is normalized with an incorrect HPO term ID
  • False Negative: a span mentioning an HPO term labeled and normalized by an annotator but not detected OR incorrectly normalized by a system

Terms and Conditions

By submitting results to this competition, you consent to the public release of your scores at the BioCreative workshop and in the associated proceedings, at the task organizers' discretion. Scores may include, but are not limited to, automatic and manual quantitative judgements, qualitative judgements, and such other metrics as the task organizers see fit. You accept that the ultimate decision of metric choice and score value is that of the task organizers.

You further agree that the task organizers are under no obligation to release scores and that scores may be withheld if it is the task organizers' judgement that the submission was incomplete, erroneous, deceptive, or violated the letter or spirit of the competition's rules. Inclusion of a submission's scores is not an endorsement of a team or individual's submission, system, or science.

You further agree that your system may be named according to the team name provided at the time of submission, or to a suitable shorthand as determined by the task organizers.

You further agree to submit and present a short paper describing your system during the workshop.

You agree not to redistribute the training and test data except in the manner prescribed by its licence.

Challenges

Both steps, the extraction and the normalization, are particularly difficult on dysmorphology physical examinations given the current state-of-the-art of natural language processing.

Extraction. This step is challenging due to the descriptive style of the examinations and their polarity. The observations are short reports where, for conciseness, the span of a finding can be disjoint or overlapping with the span of another finding. The previous observation is an example of overlapping findings, with the span palpebral fissures contributing to both HP:0000637 and HP:0000494 terms. For disjoint findings, i.e. findings defined with non-consecutive segments of text, consider the term Short nasal bridge - HP:0003194 in the observation NOSE: Short, wide nasal bridge. Anteverted nares. Designed extractors should go beyond the standard sequence labeling approach which, designed to extract contiguous and mutually-exclusive named entities, fails to capture the disjoint and overlapping terms. As an additional challenge, the extractor should also resolve the polarity of the findings, that is, automatically detecting and ignoring normal findings, only returning the key positive findings.

Normalization. This step is also challenging, both due to the large scale of the HPO ontology and its incompleteness. Standard strategies for multi-label classification are designed to assign small sets of classes to input instances. However, to be successful in our task, a normalizer should adapt traditional strategies to assign one term from among the 17,000 terms in the HPO to each finding detected in an observation. This must frequently occur without supervision since our training set does not provide examples of use for all terms in the HPO. Furthermore, while specifically designed for human genetics, and constantly improving, the HPO does not have standardized levels of term detail. As a consequence, a key finding may need to be matched with a close ancestor in the hierarchy of the ontology, making the strict matching strategy inefficient since the string of the ancestor in the HPO will be different from the string of the key finding in the observation. For example, there exists both Naevus flammeus of the eyelid - HP:0010733 and Nevus flammeus of the forehead - HP:0007413, but no term for the nose, leaving only generic Nevus flammeus - HP:0001052 to normalize this abnormality of the nose when it is mentioned in an observation.

Registration

This task is a part of a bigger competition: BioCreative VIII. To register, please follow the link "team registration page" in the section TEAM REGISTRATION of the BioCreative VIII page.

The registration is free but required to access the training and evaluation data. If you have any questions, please contact Davy Weissenbacher

Task organizers

• Graciela Gonzalez-Hernandez, Cedars-Sinai Medical Center, USA
• Ian M. Campbell, Children's Hospital of Philadelphia, USA
• Davy Weissenbacher, Cedars-Sinai Medical Center, USA
• Xinwei Zhao, Children's Hospital of Philadelphia, USA

The results of the task will be released during the BioCreative VIII workshop and published in the proceedings of the event.