Competition

OOD-CV Challenge 2023 (Classification Track - ImageNet-1k leaderboard)

Secret url: https://codalab.lisn.upsaclay.fr/competitions/14068?secret_key=b533d059-0b6d-4b81-88bd-cc7984b977b3
Organized by ROBIN - Current server time: April 3, 2025, 12:44 a.m. UTC

First phase

Phase-1: Development
June 20, 2023, midnight UTC

End

Competition Ends
Sept. 13, 2023, midnight UTC

OOD-CV Challenge (Classification Track)

The goal of our OOD-CV benchmark is to diagnose robustness of computer vision models to out-of-distribution shifts in the data. To achieve this goal, the benchmark consists of a fixed training set with 10 object categories (aeroplane, bus, car, train, boat, bicycle, motorplane, chair, dining table, sofa) from the PASCAL VOC 2012 and ImageNet datasets. Note that only our provided training data can be used to train a model and using outside training data is not allowed. This restriction enables us to design the test set such that each test example is subject to an out-of-distribution shift in one specific nuisance w.r.t. the training data, such as the object's 3D pose, shape, texture, context, the weather, and occlusion.

The OOD-CV challenge has three tracks. Each track will focus on one computer vision task, including image classification, object detection and 3D pose estimation.

Evaluation

The task of this track is Image Classification. The categories are as below:

  • aeroplane
  • bicycle
  • boat
  • bus
  • car
  • chair
  • diningtable
  • motorbike
  • sofa
  • train

Phase 1

Participants are required to submit a zip file with 1 csv files containing predictions on the test set.

The expected directory structure is

  • random_zip_name.zip
  • |-- results.csv

A valid sample submission is provided.

The submissions are evaluated using the mean value of Top-1 accuracy on each nuisance.

See here for more details.

Phase 2

Participants are required to submit a zip file with 1 csv files containing predictions on the test set.

The expected directory structure is

  • random_zip_name.zip
  • |-- results.csv

A valid sample submission is provided.

The submissions are evaluated using the mean value of Top-1 accuracy on each nuisance.

See here for more details.

 

 

Please note that this leaderboard is for ImageNet-1k pretrained models only, only model pretrained on ImageNet-1k can be used.

Terms

For phase-1, you may submit 5 submissions every day and 100 in total.

For phase-2, you may submit 3 submissions every day and 15 in total

See here for more details.

Download Size (mb) Phase
Starting Kit 0.037 #1 Phase-1: Development

Phase-1: Development

Start: June 20, 2023, midnight

Description: Directly submit results on the test set for Phase-1.

Phase-2: Final

Start: Sept. 5, 2023, noon

Description: Directly submit results on the test set for Phase-2.

Competition Ends

Sept. 13, 2023, midnight

You must be logged in to participate in competitions.

Sign In