Competition

NTIRE 2023: 360° Omnidirectional Super Resolution - Track 2 Video

Organized by chongmou - Current server time: March 29, 2025, 9:05 p.m. UTC

Previous

Testing
March 14, 2023, 5:59 p.m. UTC

Current

Development
Jan. 30, 2023, 11:59 p.m. UTC

End

Competition Ends
March 21, 2023, 11:59 p.m. UTC

NTIRE Workshop and Challenges @ CVPR 2023

 

360° Omnidirectional Video Super-Resolution Challenge

Homepage and GitHub Repo：NTIRE 2023: 360° Super-Resolution Challenge

Important dates

  • 2023.01.30 Release of train data (input and output) and validation data (inputs only)
  • 2023.01.30 Validation server online
  • 2023.03.14 Final test data release (inputs only)
  • 2023.03.20 Test output results submission deadline
  • 2023.03.20 Fact sheets and code/executable submission deadline
  • 2023.03.22 Preliminary test results release to the participants
  • 2023.03.31 Paper submission deadline for entries from the challenge
  • 2023.06.18 NTIRE workshop and challenges, results, and award ceremony (CVPR 2023, Vancouver)

 

Introduction

The 360° or omnidirectional images/videos can provide an immersive and interactive experience and have received much research attention with the popularity of AR/VR applications. Unlike planar images/videos that have a narrow field of view (FoV), 360° images/videos can represent the whole scene in all directions. However, 360° images/videos suffer from the lower angular resolution problem since they are captured by the fisheye lens with the same sensor size for capturing planar images. Although the whole 360° images/videos are of high resolution, the details are not satisfying. In many application scenarios, increasing the resolution of 360° images/videos is highly demanded to achieve higher perceptual quality and boost the performance of downstream tasks.

Recently, considerable success has been achieved in the image and video super-resolution (SR) task with the development of deep learning-based methods. Although 360° images/videos are often transformed into 2D planar representations by preserving omnidirectional information in practice, like equirectangular projection (ERP) and cube map projection (CMP), existing super-resolution methods still cannot be directly applied to 360° images/videos due to the distortions introduced by the projections. As for videos, the temporal relationships in a 360° video should be further considered since it is different from that in an ordinary 2D video. Therefore, how to effectively super-resolve 360° image/video by considering these characteristics remains challenging.

In this challenge, we aim to establish high-quality benchmarks for 360° image and video SR and expect to further highlight the challenges and research problems. This challenge can provide an opportunity for researchers to work together to show their insights and novel algorithms, significantly promoting the development of 360° image and video SR tasks.

Challenge overview

The 8th edition of NTIRE: New Trends in Image Restoration and Enhancement workshop will be held on June 18th, 2023 in conjunction with CVPR 2023.

This challenge aims to reconstruct high-resolution (HR) 360° images/videos from degraded low-resolution (LR) counterparts.

Only the training and validation sets will be released during the first phase (model construction period), and the HR and LR 360° image or video pairs are available for different tracks (360° image SR challenge and 360° video SR challenge). The participants can design their methods by considering the characteristics of 360° images/videos. Then these models can be trained on the training set and evaluated on the validation set. Note that the participants can use additional data, and the model size is not restricted.

During the second phase (testing period), the testing set containing only LR 360° images/videos will be released. The participants can super-resolve the testing LR images/videos with their trained models. The super-resolved results should be submitted by the participants and then evaluated by the organizers with the quantitative metrics.

The top-ranked participants will be awarded and invited to follow the CVPR submission guide for workshops to describe their solutions and to submit to the associated NTIRE workshop at CVPR 2023.

More details are found in the data section of the competition.

This challenge has two tracks:

  • Image track: https://codalab.lisn.upsaclay.fr/competitions/10319
  • Video track: https://codalab.lisn.upsaclay.fr/competitions/10371

and this track focuses on 360° video SR. 

Track 2：360° Omnidirectional Video Super-Resolution (X4)

  • Dataset - ODV360

    To rectify the lack of high-quality video datasets in the community of omnidirectional video super-resolution, we create a new high-resolution (4K-8K) 360° video dataset, including two parts:

    (1) 90 videos collected from YouTube and public 360° video dataset

    These videos are carefully selected and have high quality to be used for restoration. All videos have the license of Creative Commons Attribution license (reuse allowed), and our dataset is used only for academic and research proposes

    (2) 160 videos collected by ourselves with Insta360 cameras

    The cameras we use include Insta 360 X2 and Insta 360 ONE RS. They can capture high-resolution (5.7K) omnidirectional videos.
    These collected omnidirectional videos cover a large range of diversity, and the video contents vary indoors and outdoors. To facilitate the use of these videos for research, we downsample the original videos into 2K resolution (2160x1080) by OpenCV. The number of frames per video is fixed at about 100. We randomly divide these videos into training, validation, and testing sets, as shown in the following table.

 

  Training Validation Testing All
Number 210 20 20 250
Storage
 GT(59G)+LR(4.9G)
 GT(5.3G)+LR(446M)
 GT(5.7G)+LR(485M)
 75.8G

         You can download it from the following link:

         Google Drive：https://drive.google.com/drive/folders/1lDIxTahDXQ5w5x_UZySX2NOes_ZoNztN

         腾讯微云：https://share.weiyun.com/8zd7X0TZ

  • Settings
    In the process of generating low-resolution videos, we consider two main factors, i.e., downsampling and video compression. For downsampling, we apply the same pipeline as track 1. For video compression, we use the H.264 codec rules in FFMPEG to generate the compressed video frames.
    Note that we do not provide degradation scripts during this challenge to avoid potential testing data leakage.
  • Metrics
    We evaluate the super-resolved 360° video frames by comparing them to the ground truth HR ERP frames. To measure the fidelity, we adopt the widely used Weighted-to-Spherically-uniform Peak Signal to Noise Ratio (WS-PSNR) as the quantitative evaluation metric.

 

360° Omnidirectional Video Super-Resolution Challenge

 

Metrics

We evaluate the super-resolved 360° video frames by comparing them to the ground truth HR ERP frames. To measure the fidelity, we adopt the widely used Weighted-to-Spherically-uniform Peak Signal to Noise Ratio (WS-PSNR) as the quantitative evaluation metric.

The final results are ranked by WS-PSNR calculated in the RGB domain.

 

Submission

During the first phase (model construction period), the participants can evaluate the model's performance with the provided evaluation script. During the second phase (testing period), the participants should submit the super-resolved LR videos in the testing set of ODV360. 

 

In order to reduce data transfer time and upload size, we only evaluate 10 frames per video. Please use the following code to process the evaluation results:

import cv2
import os
if __name__ == '__main__':
    path = ''
    path_save = ''
    folder_list = os.listdir(path)
    for folder in folder_list:
    if os.path.exists(os.path.join(path_save,folder))==False:
        os.makedirs(os.path.join(path_save,folder))
        im_list = os.listdir(os.path.join(path,folder))
        im_list.sort()
        im_list = im_list[::10]
        for name in im_list:
            im = cv2.imread(os.path.join(path,folder,name))
            cv2.imwrite(os.path.join(path_save,folder,name), im)

When submitting, please create a ZIP archive that contains all SR videos with the same names of the corresponding LR videos. The name of each frame should also be consistent. Note that all videos should be in the root of the archive. A 'readme.txt' file also should be included in the ZIP file, containing the following content:

Runtime: 
CPU/GPU: (0/1)
Data: 
Other: 

 

After the testing phase, the participants will email the fact sheet and source code to the official submission account: osr360@outlook.com. The participants should ensure the summited codes can reproduce the submitted testing results. 

New Trends in Image Restoration and Enhancement (NTIRE) workshop and challenge on 360° Omnidirectional Super Resolution @ CVPR 2023

 

These are the official rules (terms and conditions) that govern how the NTIRE challenge on example-based 360° Omnidirectional Super Resolution 2023 will operate. This challenge will be simply referred to as the "challenge" or the "contest" throughout the remaining part of these rules and may be named as "NTIRE" or "360SR" benchmark, challenge, or contest, elsewhere (our webpage, our documentation, other publications).

In these rules, "we", "our", and "us" refer to the organizers (Mingdeng Cao, Chong Mou, Fanghua Yu, Xintao Wang, Yinqiang Zheng, Jian Zhang, Chao Dong, Gen Li, Ying Shan, Radu Timofte) of NTIRE challenge and "you" and "yourself" refer to an eligible contest participant.

Note that these official rules can change during the contest until the start of the final phase. If at any point during the contest, the registered participant considers that can not anymore meet the eligibility criteria or does not agree with the changes in the official terms and conditions then it is the responsibility of the participant to send an email to the organizers such that to be removed from all the records. Once the contest is over no change is possible in the status of the registered participants and their entries.

 

The terms and conditions are inspired by and use verbatim text from the `Terms and conditions' of ChaLearn Looking at People Challenges and of the NTIRE 2017, 2018, 2019, 2020, 2021, and 2022 challenges and of the AIM 2019, 2020, 2021, and 2022 challenges .

NTIRE Workshop and Challenges @ CVPR 2023

 

Organizers

 

The NTIRE challenge on 360° Omnidirectional Super Resolution is organized jointly with the NTIRE 2023 workshop.  The results of the challenge will be published at NTIRE 2023 workshop and in the CVPR 2023 Workshops proceedings.

 

Organizers:
Mingdeng Cao, Chong Mou, Fanghua Yu,
Xintao Wang, Yinqiang Zheng, Jian Zhang,
Chao Dong, Gen Li, Ying Shan, Radu Timofte

 

More information about NTIRE workshop and challenge organizers is available here: https://cvlai.net/ntire/2023/

Development

Start: Jan. 30, 2023, 11:59 p.m.

Testing

Start: March 14, 2023, 5:59 p.m.

Competition Ends

March 21, 2023, 11:59 p.m.

You must be logged in to participate in competitions.

Sign In
# Username Score
1 cui666 25.89
2 Spider-Man 25.41
3 PixelBE 24.75