Competition

Evaluation Criteria

Data Format

The submission zipped file should contain all test sequences in separate folders. Each test sequence folder should contain a folder named "predictions" which contains the predictions for that sequence. The predictions should be in the form of text files with the same name as the corresponding image file. For each .txt file, each row is the motion label for the corresponding point of the LiDAR scan (motion label: 0 for non-moving, 1 for moving).

    sample_submission.zip
    ├── 05
    │   └── predictions
    │         ├ 000000.txt
    │         ├ 000001.txt
    │         ├ ...
    ├── 06
    │   └── predictions
    │         ├ 000000.txt
    │         ├ 000001.txt
    │         ├ ...
    ├── 07
    .
    .
    .
    └── 09

Evaluation Metric To assess the labeling performance, we rely on the commonly applied Jaccard Index or intersection-over-union (mIoU) metric over moving points on all test sequences.

Terms and Conditions

Submission Policies

We adopt similar submission policies as SemanticKITTI: Moving Object Segmentation

Only the training set is provided for learning the parameters of the algorithms. The test set should be used only for reporting the final results compared to other approaches - it must not be used in any way to train or tune systems, for example, by evaluating multiple parameters or feature choices and reporting the best results obtained. Thus, we impose an upper limit (currently 10 attempts) on the number of submissions. It is the participant's responsibility to divide the training set into proper training and validation splits, e.g., we use sequence 04 for validation. The tuned algorithms should then be run - ideally - only once on the test data and the results of the test set should not be used to adapt the approach.

The evaluation server may not be used for parameter tuning since we rely here on a shared resource that is provided by the Codalab team and its sponsors. We ask each participant to upload the final results of their algorithm/paper submission only once to the server and perform all other experiments on the validation set. If participants would like to report results in their papers for multiple versions of their algorithm (e.g., parameters or features), this must be done on the validation data and only the best performing setting of the novel method may be submitted for evaluation to our server. If comparisons to baselines from third parties (which have not been evaluated on the benchmark website) are desired, please contact us for a discussion.

Important note:
It is NOT allowed to register multiple times to the server using different email addresses. We are actively monitoring submissions and we will revoke access and delete submissions. When registering with Codalab, we ask all participants to use a unique institutional email address (e.g., .edu) or company email address. We will not approve email addresses from free email services anymore (e.g., gmail.com, hotmail.com, qq.com).