Competition

We will use partial Average Precision (pAP) [1] as the main evaluation measure. It is a rank-based measure that integrates partial matching to give credit to a QA system that may retrieve an answer that is not necessarily at the first rank and/or \emph{partially} (i.e., not exactly) match one of the gold answers. Moreover, pAP can be used in evaluating questions that may have one or more answers in the accompanying passage. This makes pAP more suitable to the RC task of Qur'an QA 2023 than partial Reciprocal Rank (pRR) [2], which was the main evaluation measure for Qur'an QA 2022, because participating systems in the latter task were only required to return any answer to a given question even if it has more than one answer in the accompanying passage.  

Similar to the PR task, the no-answer cases will be handled simply by giving full credit to ``no answers'' system output and zero otherwise. 

To get an overall evaluation score, the measure is averaged over all questions.

Terms and Conditions

The QRCD (Qur'anic Reading Comprehension Dataset) is distributed under the CC BY-NC-ND 4.0 License https://creativecommons.org/licenses/by-nc-nd/4.0/legalcode

For a human-readable summary of (and not a substitute for) the above CC BY-NC-ND 4.0 License, please refer to https://creativecommons.org/licenses/by-nc-nd/4.0/

Terms & Conditions:

• It is strictly prohibited to make any changes to the QRCD dataset, given that the answers to the questions were carefully extracted from the Holy Qur'an and then annotated by Qur'an Scholars.
• Considering that there are different schools of thought in Islam, the QRCD dataset represents a sample and not all schools of thought.
• Any suggestions for adding or refining the answers to existing questions (or adding new questions and answers) can be directed to the Qur'an QA 2023 organizers, who in turn must solicit the feedback of Qur'an scholars before effecting any updates on the QRCD dataset.
• We note that answer spans can only be extracted from their corresponding verse-based direct answers in the AyaTEC dataset. Only Qur'an scholars can decide if a verse-based answer represents a direct or indirect answer to a given question. For a formal definition of a direct and indirect answer, refer to the AyaTEC paper (p 11).