Evaluation dataset

Please refer to https://www.omnilabel.org/dataset/download for the detailed description of how to download and set up the data.

Training data for Track C

• Allowed training datasets: Any

• Allowed pre-trained models: Any

Disallowed: Do not use the validation/test sets of COCO, Objects-365 or OpenImages for training. Also, we discourage the use of the OmniLabel validation set for training/fine-tuning.

How to submit your results

Submission format

1. Create answer.json file in the following format:

[

    {

        image_id        ... the image id this predicted box belongs to

        bbox            ... the bounding box coordinates of the object (x,y,w,h)

        description_ids ... list of description IDs that refer to this object

        scores          ... list of confidences, one for each description

    },

    ...

]

2. Create a ZIP archive by compressing answer.json file. (See an example submission file: example_dummy_result.zip)

Task

Please refer to https://www.omnilabel.org/task for the detailed description of the task

Inputs:

Each sample in the dataset is a pair of image I and label space list D. Each image I has a unique image_id. Similarly, the label space D is a list of object descriptions, each coming with a unique description_id and a text, e.g., D = [{id=321, text="Person"}, {id=223, text="Donuts"}, {id=4321, text="Chocolate donut"}, {id=12, text="Donut with green glaze"}].

Outputs:

The expected output of a model M is a set of triplets (bbox, score, description_id). Each triplet consisting of a bounding box, a confidence score, and an index linking the prediction to an object description in D. A bounding box bbox = (x,y,w,h) consists of 4 coordinates in the image space and defines the extent of an object. The confidence score is a real-valued scalar expressing the confidence in the model's prediction. Finally, the index description_id points to an ID in the label space D and indicates that the box is described by the corresponding text. Note that one object in the image may be referred to by multiple object descriptions (e.g., “person” and “woman in red shirt”), in which case the output should be one bounding box which points to two descriptions.

Evaluation metric

To evaluate a model M on our task, we can follow the basic evaluation protocol of standard object detection with the Average Precision (AP) metric. This metric computes precision-recall curves at various Intersection-over-Union (IoU) thresholds to evaluate both classification and localization ability of a model M. Compared to traditional object detection, we adjust the evaluation protocol to account for the complex, free-form text object descriptions, which are virtually infinite in size:

• The grouping of the label space is different. Instead of computing AP per-class and taking the mean, we compute AP for different, and potentially overlapping groups of the label space. One group computes AP for all descriptions, other groups only for free-form object descriptions of different length, etc.

• One predicted box can be matched to multiple object descriptions, e.g., "person" and "woman in red shirt"

Our evaluation toolkit is available on github

Terms and Conditions

The participants own the copyright of their source code and trained models.