Evaluation Criteria

Submissions are evaluated on the test set. We report mean Average Precision (mAP) for verbs, nouns and actions at different IoU thresholds (0.1, 0.2, 0.3, 0.4, 0.5) and average mAP across IoU thresholds on the overall test set:

We consider mAP as implemented in [1]. Methods are ranked by average action mAP.

Note that on 01/02/2024 the evaluatiuon code has been updated to fix some minor errors. As a result re-evaluating the same submission may lead to minor changes in performance evaluation.

References

Terms and Conditions

• You agree to us storing your submission results for evaluation purposes.
• You agree that if you place in the top-10 at the end of the challenge you will submit your code so that we can verify that you have not cheated.
• You agree not to distribute the EPIC-KITCHENS-100 dataset without prior written permission.

Submissions

To submit your results to the leaderboard you must construct a submission zip file containing a single file test.json containing the model’s results on the test set. This file should follow format detailed in the subsequent section.

JSON Submission Format

The JSON submission format is composed of a single JSON object containing entries for every detected action in all the videos of the test set. Specifically, the JSON file should contain:

• a 'version' property, set to '0.2'
• a 'challenge' property, set to 'action_detection'
• a set of sls properties (see the Supervision Levels Scale (SLS) page for more details):
  • sls_pt: SLS Pretraining level.
  • sls_tl: SLS Training Labels level.
  • sls_td: SLS Training Data level.
• a 'results' object containing entries for every video in the test set (e.g . 'P01_101').

Each video entry is a list of objects describing each detected action. Each of these objects should contain:

• a 'verb' property, reporting the detected verb class (e.g., 1).
• a 'noun' property, reporting the detected noun class (e.g., 34).
• a 'action' property, reporting the detected action class in the format '<verb_class>,<noun_class>' (e.g., ‘1,34’). The predicted action may differ from the pair of predicted verb and noun classes.
• a 'score' property, reporting the confidence score of the prediction (e.g., 0.78).
• a 'segment' property, which is a list containing the starting and ending timestamps of the detected action in seconds (e.g., [6.13, 9.20]).

{
  "version": "0.2",
  "challenge": "action_detection",
  "sls_pt": -1,
  "sls_tl": -1,
  "sls_td": -1,
  "results": {
    "P26_122": [
            {
                "verb": 0,
                "noun": 16,
                "action": "0,16",
                "score": 0.7398802638053894,
                "segment": [
                    25.223000000000004,
                    32.86633333333334
                ]
            },
            ...
            {
                "verb": 0,
                "noun": 58,
                "action": "0,58",
                "score": 0.0001102862200564619,
                "segment": [
                    292.73966666666669,
                    300.38300000000006
                ]
            }
        ],
    ...
    "P36_102": [
            {
                "verb": 9,
                "noun": 27,
                "action": "9,27",
                "score": 0.8049795031547546,
                "segment": [
                    599.3945,
                    613.28775
                ]
            },
            ...
            {
                "verb": 17,
                "noun": 65,
                "action": "17,65",
                "score": 0.0006565209107522163,
                "segment": [
                    567.6384999999999,
                    573.59275
                ]
            }
        ]
    }
}

You can provide scores and timestamps in any float format that numpy is capable of reading (i.e. you do not need to stick to a given number of decimal places).

Submission archive

To upload your results to CodaLab you have to zip the test file into a flat zip archive (it can’t be inside a folder within the archive).

You can create a flat archive using the command providing the JSON file is in your current directory.

$ zip -j my-submission.zip test.json