Welcome to the 5LSM0 Competition!

Throughout the lectures, various topics concerning Neural Networks (NNs) for computer vision have been covered. In this competition, we challenge you to put your newly acquired skills to the test by working with a real-world dataset.

You will have the opportunity to work with the CityScapes dataset, a large-scale collection of images capturing various aspects of urban environments, such as street scenes, buildings, vehicles, and more. By participating in this competition, you will gain valuable experience in applying your knowledge to real-world problems and improve your skills in computer vision.

This competition will focus on multiple problems that are relevant but often under-addressed in a research setting, however, have proven to be crucial for real-world applications. For each of these problems we have created a specific benchmarking test set, related to CityScapes.

Fig 1. Here a sample from the dataset is shown, together with the semantic label map of the sample.

Baseline

Before diving into the main benchmarks, you must first create a baseline submission. This step ensures that your workflow is correctly set up, your model training and evaluation pipeline is functional, and your submissions adhere to the competition requirements. The baseline phase is critical to verify that you are on the right track. It also helps establish a reference point for your subsequent improvements. Please note that there is a strict deadline for submitting the baseline results to ensure everyone progresses smoothly toward the final assignment.

Benchmarks

After providing the first baseline results your can chose a specific direction for your final assignment. The challenge consists of four benchmarks, each addressing a different aspect:

1. Peak Performance

This is the first benchmark and will also sever as a baseline submission. Generally, this subtask is the focus of computer vision challenges. This benchmark assesses the model's performance on a clean test set. Evaluation is based on the model's ability to achieve the highest segmentation scores on a designated test set.

2. Robustness

The robustness benchmark evaluates the model's resilience against various conditions, such as changes in lighting, weather, or decreased image quality. Consistent performance across different scenarios is needed to excel in this category.

3. Efficiency

Efficiency is crucial for practical deployment of computer vision models. This benchmark focuses on evaluating models on their size. Often, in the presence of large amounts of data, larger models outperform their smaller counterparts. However, on edge devices, such as FPGAs, very large model can't be used. Therefore, this benchmarks focuses on reducing model size while maintaining acceptable performance.

4. Out-of-Distribution Detection

Models often encounter data that differ significantly from the training distribution, leading to unreliable predictions. This benchmark assesses the model's ability to detect out-of-distribution samples to leave out for final evaluation.

Fig 2. Some examples of challenging conditions to which the network will face in deployed in a real world setting, but are not present in the training set.

We wish you the best of luck in this competition, and we look forward to seeing your innovative solutions!