Competition

Evaluation

Submission Format

You must submit a submission bundle (a .zip archive) with a single JSON file named submission.json. This file should contain key-value pairs, in which the key is the name of the video, and the value is a item, which is the predicted class ID (an integer in the interval [0, 19]).

For example, if "video0000.mp4" is classified as "Drinking water" and the key-value pair would be "video0000.mp4": [4]. A video sample considered to be Unknown activity can be represented as "video0001.mp4": [19].

{"video0000.mp4": [3], "video0001.mp4": [7], "video0002.mp4": [2], "video0003.mp4": [1], .... }

A sample submission file has been provided in the starting kit.

Use the following command to create the submission bundle:

zip -j submission.zip submission.json

Evaluation Criteria

The final rank is based on the true positive rate.

1. True positive rate (TPR) of activities: number of true positive activity detections / total number of videos

Terms and Conditions

This challenge is organized by HEART-MET and has received funding from the European Union’s Horizon 2020 research and innovation program under grant agreement No 871252 (METRICS).

The Codalab stage of the competition consists of two phases: Development and Testing. A training and validation set are available during the Development phase, and a test dataset is available during the Testing phase. The Remote Execution stage of the competition will take place after the Testing phase.

1. It is open to the public, but requires participants to register using a Codalab account.
2. Participation is possible by individuals and teams.
3. If participating in a team, each team member must be registered as a participant (for details see Overview).
4. Each participant is only allowed to create one account.
5. Final rankings of the Codalab stage are based on the results in the Testing phase.
6. The Remote Execution stage is only open to the top 3 submission of the Testing phase.
7. Final rankings of the overall competition are based on the Remote Execution stage.
8. Participants are allowed (and encouraged) to use external data for training models.
9. It is not permitted to use the validation or test data for training models.
10. Manually labeled submissions are not permitted. Any submission which includes results which have been manually entered will be disqualified.
11. At the end of the Testing phase, the top 3 winners will be asked to submit their software, trained models and a report describing their approach to the organizers.
12. The report should contain details of the data used for training (including details of any external data used), training models and process, and inference process.
13. The datasets are released under the Creative Commons Attribution 4.0 International license.

Dataset format

The dataset consists of video clips around 5-10 seconds long, of a person performing an activity.

Activity Classes

There are a total of 20 activities in this challenge. In addition to the activities, the dataset (both training and validation/test) contains one classs, which is unknown activity

The unknown activity include videos of unrelated content (e.g. a video with no person, a person performing activities outside of the 19 classes, etc.) and videos in which the data is corrupted (e.g. blurred videos, noisy videos etc.). The corrupted videos may contain relevant activities or unrelated content.

The activity IDs and the corresponding classes contained in this dataset are listed below:

Label Format

The labels for the training set are provided as a single JSON file, consisting of key-value pairs of the video file name and the corresponding label(s). The labels are provided as a list, which may consist of one or two elements.

The list may consist of:

1. a single element consisting of an in-distribution class; e.g. [5]
2. a single element consisting of the unknown activity class; e.g. [19]

Example:

{"video0000.mp4": [2], "video0001.mp4": [19], "video0002.mp4": [5], .... }

Refer to the Evaluation section for the details.

Training, validation and test datasets

The data is split into training, validation and test datasets. The training and validation datasets are available during the Development phase, with labels provided only for the training dataset. You can evaluate your methods on the validation set by submitting your results to the Codalab server.
The test dataset will be available at the start of the Testing phase. During the test phase, you should submit results for the test set to the Codalab server.