competitions platform and its new (beta)
are powerful open source frameworks for running
scientific competitions and benchmarks, which involve result
or code submission. You can either participate
in an existing competition (or benchmark) or host one.
Most competitions hosted on such platforms are machine
learning (data science) competitions, but they are NOT
limited to this application domain. They can accommodate
any problem for which a solution can be provided in the
form of a zip archive containing a number of files to be
evaluated quantitatively by a scoring program (provided
by the organizers). The scoring program must return a
numeric score, which is displayed on a leaderboard where
the performances of participants are compared.
Codalab is more geared towards organizing limited-time
competitive events (challenges) and Codabench
is designed for collaborative benchmarking of
algorithms against sets of tasks (or vice-versa)
over a long period of time. Codabench is more flexible
than Codalab and fully backward compatible (old Codalab
challenge bundles can be uploaded to Codabench). But is
is newer and still in a beta version.
Codalab was created in 2013 as a joint venture between
Microsoft and Stanford University. Originally the vision
was to create an ecosystem for conducting computational
research in a more efficient, reproducible, and
collaborative manner, combining worksheets and
competitions. Worksheets capture complex research
pipelines in a reproducible way and create "executable
In 2014, ChaLearn joined to
co-develop Codalab competitions. Since 2015, University
Paris-Saclay is community lead of Codalab competitions,
under the leadership of Isabelle Guyon,
professor of artificial intelligence. Codalab is
administered by LISN staff,
under the direction of Anne-Catherine
Since 2019 we are developing a new version of Codalab
competitions called Codabench, to organize both
competitions and benchmarks.
Codalab is used actively in research. In 2020/2021, 600
new challenges were launched. Recent popular challenges
organized with Codalab include the AAAI
2021 Covid-19 fake news detection (over 1000
participants), the COVID-19
infection percentage estimation, the COVID-19
retweet prediction challenge, the ECCV
2020 ChaLearn LAP Fair face recognition challenge,
the 2020 DriveML
Huawei Autonomous Vehicle Challenge. Recurrent
challenges attracting hundreds of participants each year
include the SemEval challenge
series in natural language processing and
the ChaLearn LAP
challenge series in computer vision.
High profile challenges include the 2
million Euro prize of the EU, organized by the See.4C
consortium, the CIKM
AnalytiCup 2017, which attracted 493 participants,
(633 participants), the ChaLearn
AutoML challenge 2017 (687 participants), and the
Tumor Segmentation Challenge (over 4000
Since 2016, Codalab offers the possibility of
organizing machine learning challenges with code
submission. The simplest machine learning challenges
require only the submission of results, which are
compared to a solution (or key) by a scoring program.
Result submission challenges are less computationally
expensive than code submission challenges. However, they
offer less possibilities. In particular, code submission
allows conducting fair benchmarks by executing submitted
code in the same condition for all participants.
Codalab has been providing free resources for challenge
organizers. New since version 1.5: organizers can hook
up their own compute workers to the backend of Codalab
to redirect the code submissions, enabling growth to big
data competitions running at the expense of the
organizers. For very special dedicated projects, Codalab
can be customized since it is an open source project.
Codalab is used in education. For example, a class on Artificial
Neural Networks and Deep Learning 2021 with
400 students had homework on Codalab. We implemented a
software called Chagrade to
help grading homework using Codalab competitions.
Mars 2022: The new Codalab infrastructure is stable.
We have migrated the storage over a distributed Minio
(4 physical servers, each with 12 disks of 16 TB)
spread over 2 buildings for robustness,
and added 10 more GPUs to the existing 10 previous ones in the backend.
A lot of horsepower to suport Industry-strenght challenge.
Thanks for the sponsorship of région Ile-de-France, ANR,
Université Paris-Saclay, CNRS, INRIA, and ChaLearn.
The new server is faster, runs in Python3, and features a renewed organizer interface.
December 2021: Codabench (beta) is announced at
August 2020: Codalab exceeds 50,000 users,
1000 competitions (over 400 last year), and ~600
submissions per day!
May 2020: We released a new application to
help instructors use challenges in the classroom and
grade them called Chagrade.
October 2018: We are preparing a challenge
Automatic Deep Learning (AutoDL) challenging
participants to design code eliminating the need of
human expertise to choose the architecture and
hyper-parameters of deep neural networks. The
challenge is co-organized and sponsored by Google. The
protocol will be tested on Codalab in October.
September 2018: The LAL and CERN are organizing a
challenge to reconstruct particle trajectories in high
energy physics detectors. After the success of the first
phase with result submission only, a second
phase with code submission will be run on Codalab.
TrackML is an officially selected challenge of the
NIPS 2018 conference.
February 2018: 2 million Euro Big Data EU
prize powered by Codalab.
February 2018: Isabelle Guyon presents Codalab
at the newly formed Institute of Convergence DataIA
January 2018: Paris-Saclay master students
create challenges for L2 students.
January 2018: Paris-Saclay instructors create
reinforcement learning homework.
December 2017: Codalab exceeds 10000 users
with 480 competitions (145 public)
December 2017: Codalab presented at the
Challenges in Machine Learning workshop [slides].
Version 1.5 is out!
November 2017: Explore the new features:
scale up your code submission competition with your
own compute workers (full privacy, dockers); organize
RL challenges and hook up simulators providing data on
demand (with your own "ingestion program"); use the
ChaLab wizard to create competitions in minutes.