an open source platform to learn, create, collaborate through challenges

3113
Total Competitions
1128
Public Competitions
1985
Private Competitions
70069
CodaLab Website Users
111600
Competition Participants
791189
Submissions

What are Competitions and Benchmarks?

The CodaLab competitions platform and its new (beta) version Codabench are powerful open source frameworks for running scientific competitions and benchmarks, which involve result or code submission. You can either participate in an existing competition (or benchmark) or host one.

Most competitions hosted on such platforms are machine learning (data science) competitions, but they are NOT limited to this application domain. They can accommodate any problem for which a solution can be provided in the form of a zip archive containing a number of files to be evaluated quantitatively by a scoring program (provided by the organizers). The scoring program must return a numeric score, which is displayed on a leaderboard where the performances of participants are compared.

Codalab is more geared towards organizing limited-time competitive events (challenges) and Codabench is designed for collaborative benchmarking of algorithms against sets of tasks (or vice-versa) over a long period of time. Codabench is more flexible than Codalab and fully backward compatible (old Codalab challenge bundles can be uploaded to Codabench). But is is newer and still in a beta version.

History of Codalab and Codabench

Codalab was created in 2013 as a joint venture between Microsoft and Stanford University. Originally the vision was to create an ecosystem for conducting computational research in a more efficient, reproducible, and collaborative manner, combining worksheets and competitions. Worksheets capture complex research pipelines in a reproducible way and create "executable papers".

In 2014, ChaLearn joined to co-develop Codalab competitions. Since 2015, University Paris-Saclay is community lead of Codalab competitions, under the leadership of Isabelle Guyon, professor of artificial intelligence. Codalab is administered by LISN staff, under the direction of Anne-Catherine Letournel.

Since 2019 we are developing a new version of Codalab competitions called Codabench, to organize both competitions and benchmarks.

Research and Education

Codalab is used actively in research. In 2020/2021, 600 new challenges were launched. Recent popular challenges organized with Codalab include the AAAI 2021 Covid-19 fake news detection (over 1000 participants), the COVID-19 infection percentage estimation, the  COVID-19 retweet prediction challenge, the ECCV 2020 ChaLearn LAP Fair face recognition challenge, the 2020 DriveML Huawei Autonomous Vehicle Challenge. Recurrent challenges attracting hundreds of participants each year include the SemEval challenge series in natural language processing and the ChaLearn LAP challenge series  in computer vision. High profile challenges include the 2 million Euro prize of the EU, organized by the See.4C consortium, the CIKM AnalytiCup 2017, which attracted 493 participants, MSCOCO (633 participants), the ChaLearn AutoML challenge 2017 (687 participants), and the Liver Tumor Segmentation Challenge (over 4000 participants).

Since 2016, Codalab offers the possibility of organizing machine learning challenges with code submission. The simplest machine learning challenges require only the submission of results, which are compared to a solution (or key) by a scoring program. Result submission challenges are less computationally expensive than code submission challenges. However, they offer less possibilities. In particular, code submission allows conducting fair benchmarks by executing submitted code in the same condition for all participants.

Codalab has been providing free resources for challenge organizers. New since version 1.5: organizers can hook up their own compute workers to the backend of Codalab to redirect the code submissions, enabling growth to big data competitions running at the expense of the organizers. For very special dedicated projects, Codalab can be customized since it is an open source project.

Codalab is used in education. For example, a class on Artificial Neural Networks and Deep Learning 2021 with 400 students had homework on Codalab. We implemented a software called Chagrade to help grading homework using Codalab competitions.

News

New server

Mars 2022: The new Codalab infrastructure is stable. We have migrated the storage over a distributed Minio (4 physical servers, each with 12 disks of 16 TB) spread over 2 buildings for robustness, and added 10 more GPUs to the existing 10 previous ones in the backend. A lot of horsepower to support Industry-strenght challenge. Thanks for the sponsorship of région Ile-de-France, ANR, Université Paris-Saclay, CNRS, INRIA, and ChaLearn. The new server is faster, runs in Python3, and features a renewed organizer interface.

Codabench

December 2021: Codabench (beta) is announced at NeurIPS 2021.

Codalab statistics

August 2020: Codalab exceeds 50,000 users, 1000 competitions (over 400 last year), and ~600 submissions per day!

L2RPN

July 2020: We launched a new Learning to Run a Power Network competition, in collaboration with ChaLearn and RTE. We have a robustness and an adaptability track. This is an NeurIPS 2020 competition.

Chagrade

May 2020: We released a new application to help instructors use challenges in the classroom and grade them called Chagrade.

AutoDL

April 2020: The NeurIPS AutoDL challenge ended. But the series of challenges on Automated Deep Learning, in collaboration with ChaLearn, Google Zurich, and 4Paradigm continues with AutoSeries and AutoGraph.

Data Science Africa 2019

June 2019: We organized a data science bootcamp at Data Science Africa 2019 in the form of a challenge to detect Malaria parasites in microscope images.

L2RPN

May 2019: We launched the first Learning to Run a Power Network competition, in collaboration with ChaLearn and RTE. This is an IJCNN competition.

AutoDL

October 2018: We are preparing a challenge on Automatic Deep Learning (AutoDL) challenging participants to design code eliminating the need of human expertise to choose the architecture and hyper-parameters of deep neural networks. The challenge is co-organized and sponsored by Google. The protocol will be tested on Codalab in October.

TrackML

September 2018: The LAL and CERN are organizing a challenge to reconstruct particle trajectories in high energy physics detectors. After the success of the first phase with result submission only, a second phase with code submission will be run on Codalab. TrackML is an officially selected challenge of the NIPS 2018 conference.

AutoML3

August 2018: Codalab is proud to host the third challenge on Automatic Machine Learning: Lifelong Machine Learning with drift. AutoML3 is an officially selected challenge of the NIPS 2018 conference.

See.4C

February 2018: 2 million Euro Big Data EU prize powered by Codalab.

DataIA

February 2018: Isabelle Guyon presents Codalab at the newly formed Institute of Convergence DataIA

Student Projects

January 2018: Paris-Saclay master students create challenges for L2 students.

Homework

January 2018: Paris-Saclay instructors create reinforcement learning homework.

10,000 Users

December 2017: Codalab exceeds 10000 users with 480 competitions (145 public)

CiML workshop

December 2017: Codalab presented at the Challenges in Machine Learning workshop [slides].

Version 1.5 is out!

November 2017: Explore the new features: scale up your code submission competition with your own compute workers (full privacy, dockers); organize RL challenges and hook up simulators providing data on demand (with your own "ingestion program"); use the ChaLab wizard to create competitions in minutes.