Metrics

The metrics calculation code can be found here.

Task 1: Quadratic-weighted Kappa (QWK), F1 score, Specificity.

Task 2: Dice Similarity Coefficient (DSC).

Task 3: Coefficient of determination R-squared, Mean Absolute Error (MAE).

Ranking methods

Task 1: First, separate metric scores are computed for quadratic-weighted kappa, F1 score and specificity on all test cases. Second, the ranking score is obtained by averaging the scores of all metrics. The achieved quadratic-weighted Kappa will be used as a tie-break.

Task 2: The dice similarity coefficient is calculated for each class independently and then results are averaged for the final ranking. In case of a tie, the recall and precision will be used for auxiliary ranking.

Task 3: First, separate rankings are computed for R-squared and mean absolute error on all test cases. Second, the mean ranking of each team is computed as the average of the two rankings. The achieved R-squared will be used as a tie-break.

Validation phase

During this phase, participants will have the opportunity to evaluate their methods on a validation set and ensure that their code submission is in the correct format. Please note that this phase does not count towards final rankings. Once participants complete their submission, the scoring program will run automatically, and the evaluation results will be displayed on the leaderboard. The best results among multiple submissions will be displayed on the leaderboard.

Test phase

During this phase, the submissions from each team will be evaluated automatically on the test set. The leaderboard will be hidden and not visible to the participants. Each team is allowed a maximum of four submissions, and the best result will be taken as the final challenge result. Once the challenge is over, we will publicly release the evaluation results on the test set, as well as the rankings of each team. Please note that only submissions from TEAM LEADER are eligible for the final ranking.

Future test phase

After the competition, participants are given the opportunity to evaluate their algorithms by submitting the results of the test set for evaluation. Please note that the Future Test Phase is not part of MMAC, and the submissions in this phase will not affect the final winners of the competition. The winners will be determined solely based on the results from the Test Phase.

Challenge rules

Participants of this challenge acknowledge that they have read and agree to the following rules:

News

[2023.08.28] Please note that only teams or individuals submitting challenge papers are eligible for the final rankings.

[2023.08.28] The validation set for each task has been released. The download link can be found in the "Get Data" section of the "participate" tab. At present, there are no plans to release the test set to the public. If participating teams are interested in evaluating the model's performance on the test set, including ablation experiments and other analyses, we kindly request that you submit and validate your mothods in the Future test phase.

[2023.08.16] In light of a ten-day extension to the test phase deadline, we have considered the situation of certain teams depleting their allotted number of submissions. For participants who have previously submitted during this phase, we now offer the opportunity to contact us via our official email address. They can request the removal of their prior submissions, enabling them to make new submissions within the test phase.

[2023.08.14] In response to valuable feedback from participants, and with the aim of allowing ample time for challenge submissions, we have decided to extend the submission deadline. The new deadline for test set submissions August 25, 2023. The new deadline for paper submissions is September 15, 2023.

[2023.08.14] In relation to Tasks 2 and 3, we have noted that several teams successfully submitted their results during the validation phase but have yet to submit during the subsequent test phase. We strongly recommend that these teams, as well as all other participating teams, make an active submission to win the challenge. Your active participation not only enhances your chances of winning but also contributes to the overall competition's success.

[2023.08.14] We have thoroughly refined the instructions for paper submissions. For detailed guidelines, please refer to the updated "Paper submission" section in "Terms and Conditions" tab.

[2023.07.25] Based on the entries and feedback from the validation phase, the code execution time limit for the test phase is 14,400 seconds.

[2023.07.21] Python Packages has been updated and will remain unchanged. New installation requests for the Python package will no longer be accepted.

[2023.07.18] The limit on the number of submissions in validation phase has been increased to 400.

[2023.07.13] The deadline for the submission of validation sets has been extended to July 25th.

[2023.07.13] Python Packages has been updated.

[2023.07.13] To enable participants to conduct code correctness checks in the validation phase using the latest Python environment, and to ensure consistency in Python package usage between the validation phase and test phase, the deadline for Python package installation requests is July 20th.

[2023.06.25] Python Packages has been updated.

[2023.06.18] The training set for task 2 has been released. The download link can be found in the "Get Data" section of the "participate" tab.

[2023.05.31] More images have been added to the training set for task 1.

[2023.05.25] The training sets for task 1 and task 3 have been released. The download link can be found in the "Get Data" section of the "participate" tab.

[2023.04.23] The dataset has not been released yet, once released, the dataset download link will be sent to approved participants via email.

FAQ

Join the challenge

1. To get started, create a CodaLab account or log in to your existing one.
2. Once you've logged in, navigate to the "participate" tab and register for the challenge.
3. To complete your registration, you must fill out the Challenge Registration form. During this process, ensure that you download and complete the MMAC Challenge Rules Agreement Consent. Make sure to hand sign the document in the "signature" section and name the file "MMAC Challenge Rule Agreement_Your First Name_Your Last Name" before uploading it in PDF format.
4. We will review your registration request within 3 working days. If you don't receive feedback within this timeframe, please reach out to us via the official email address of the challenge.

* Note: If you plan to participate in multiple tasks, you will only need to fill out the challenge registration form once.

Teams

The challenge has ended. We have now made the dataset and algorithm code publicly available.

• MMAC dataset download. The MMAC dataset has been deposited to the Zenodo data repository (DOI 10.5281/zenodo.11025749) and can be downloaded from https://zenodo.org/records/11025749.

• Algorithm papers.  The 15 submitted algorithms - 7 for Task 1, 4 for Task 2, and 4 for Task 3 - have been detailed in 11 papers published in the Challenge Proceedings, which can be downloaded from https://link.springer.com/book/10.1007/978-3-031-54857-4.

• Evaluation codes & model weights. The evaluation codes and model weights submitted in the test phase for the 15 algorithms can be downloaded from this link. Researchers can use these codes and weights to test the performance of these algorithms on their own datasets.

• Training code. The training code for each team is available at the following GitHub link. Researchers can further develop their own algorithms based on these existing models or train the model using their own datasets.

  • Task 1 Rank 1, fdvts_mm: https://github.com/FDU-VTS/MMAC
  • Task 1 Rank 2, DGUT_luli: https://github.com/LuliDreamAI/MICCAI_TASK1
  • Task 1 Rank 3, Taco Friday: https://github.com/Retinal-Research/NN-MOBILENET
  • Task 2 Rank 1, fdvts_mm: https://github.com/FDU-VTS/MMAC
  • Task 2 Rank 2, latim: https://github.com/liyihao76/MMAC_LaTIM_Solution
  • Task 2 Rank 3, hyeonminkim0625: https://github.com/hyeonminkim0625/mmac_multitask
  • Task 3 Rank 1, latim: https://github.com/liyihao76/MMAC_LaTIM_Solution
  • Task 3 Rank 2, AIFuture: https://github.com/yeungbo/AIFUTURE4MMAC23
  • Task 3 Rank 3, Taco Friday: https://github.com/HuayuLiArizona/Prediction-of-Spherical-Equivalent-With-Vanilla-ResNet

If you use this dataset for your research, please cite our paper:

@article{qian2024competition,
title={A competition for the diagnosis of myopic maculopathy by artificial intelligence algorithms},
author={Qian, Bo and Sheng, Bin and Chen, Hao and Wang, Xiangning and Li, Tingyao and Jin, Yixiao and Guan, Zhouyu and Jiang, Zehua and Wu, Yilan and Wang, Jinyuan and others},
journal={JAMA ophthalmology},
volume={142},
number={11},
pages={1006--1015},
year={2024},
publisher={American Medical Association}
}

Organizers

Challenge chairs

• Tien Yin Wong, Tsinghua University, Beijing, China
• Bin Sheng, Shanghai Jiao Tong University, Shanghai, China

Challenge committee chairs

• Hao Chen, The Hong Kong University of Science and Technology, Hong Kong, China
• Xiangning Wang, Shanghai Jiao Tong University Affiliated Sixth People's Hospital, Shanghai, China

Challenge committee members

• Yih-Chung Tham, Singapore Eye Research Institute, Singapore National Eye Centre, Singapore
• Ching-Yu Cheng, Singapore Eye Research Institute, Singapore National Eye Centre, Singapore
• Marcus Ang, Singapore Eye Research Institute, Singapore National Eye Centre, Singapore
• Carol Y Cheung, The Chinese University of Hong Kong, Hong Kong, China
• Qiang Wu, Shanghai Jiao Tong University Affiliated Sixth People's Hospital, Shanghai, China
• Rongping Dai, Peking Union Medical College Hospital, Beijing, China
• Xinyuan Zhang, Beijing Tongren Hospital, Beijing, China
• Jie Shen, Shanghai Jiao Tong University Affiliated Sixth People's Hospital, Shanghai, China
• Feng Lu, Huazhong University of Science and Technology, Wuhan, Hubei, China
• Mingang Chen, Shangahai Development Center of Computer Software Technology, Shanghai, China
• Xiaokang Yang, Shanghai Jiao Tong University, Shanghai, China
• Yaohui Jin, Shanghai Jiao Tong University, Shanghai, China
• Haitao Song, Shanghai Jiao Tong University, Shanghai, China
• Yang Wen, Shenzhen University, Shenzhen, Guangdong, China
• Yinfeng Zheng, Zhongshan Ophthalmic Center, Sun Yat-sen University, Guangzhou, Guangdong, China
• Huating Li, Shanghai Jiao Tong University Affiliated Sixth People's Hospital, Shanghai, China
• Pheng-Ann Heng, The Chinese University of Hong Kong, Hong Kong, China
• Daniel Ting, Singapore Eye Research Institute, Singapore National Eye Centre, Singapore
• Gavin Siew Wei Tan, Singapore Eye Research Institute, Singapore National Eye Centre, Singapore
• Bo Qian, Shanghai Jiao Tong University, Shanghai, China
• Tingyao Li, Shanghai Jiao Tong University, Shanghai, China
• Gengyou Huang, Shanghai Jiao Tong University, Shanghai, China
• Zhengrui Guo, The Hong Kong University of Science and Technology, Hong Kong, China
• Tingli Chen, Huadong Sanatorium, Wuxi, Jiangsu, China
• Jia Shu, Shanghai Jiao Tong University, Shanghai, China
• Yan Zhou, Peking Union Medical College Hospital, Beijing, China

Technical support

• Xiuyuan Chen, Shanghai Jiao Tong University, Shanghai, China
• Shangmin Huang, Shanghai Jiao Tong University, Shanghai, China
• Ruixue Zhang, Zaozhuang University, Zaozhuang, Shandong, China