Night Photography Rendering Challenge

Important dates

• 07.02.2025 Release of the train data and first validation data
• 18.02.2025 Release of the second validation data
• 21.02.2025 Deadline for submission in the second validation checkpoint
• 23.02.2025 The people's choice of the second validation dataset is published
• 02.03.2025 Release of the third validation data
• 17.06.2025 NTIRE workshop and challenges, results and award ceremony (CVPR 2025, Online)
• 04.03.2025: Deadline for submission in the third validation checkpoint
• 06.03.2025: The people's choice of the third validation dataset is published
• 11.03.2025: Release of the fourth validation data
• 13.03.2025: Deadline for submission in the fourth validation checkpoint
• 15.03.2025: The people's choice of the fourth validation dataset is published
• 19.03.2025: Release of the test data
• 21.03.2025: Deadline for submission in the test checkpoint. Please note: we require docker images and .tex reports of your solutions
• 24.03.2025: Final results are released and winners of the challenge are declared

News

Final Results Released!

Dear participants,
We are excited to share the final results of the 2025 Night Photography Rendering Challenge.

A total of 10 submissions were received by the deadline. All of the submissions outperformed the baseline solution in both objective metrics and perceptual quality and thus will be included in the report.

Objective metric evaluation

The winners of objective metric evaluation category are:

• 1st place: NJUST_KMG, Nanjing University of Science and Technology, China
• 2nd place: Mialgo, Xiaomi Inc., China
• 3rd place: PSU Team, Prince Sultan University, Saudi Arabia

The detailed results are presented below.

Perceptual quality evaluation

The winners of perceptual quality evaluation category are:

• 1st place: Mialgo, Xiaomi Inc., China
• 2nd place: NJUST_KMG, Nanjing University of Science and Technology, China
• 3rd place: PSU Team, Prince Sultan University, Saudi Arabia

The detailed results are presented below.

The winners will be informed on how to receive their prizes in the upcoming days.

We sincerely congratulate the winners and thank all teams for participating in our challenge. We wish you the best of luck in all your future endeavors.

Best regards,
The NTIRE Night Photography Rendering Challenge Team

Final Testing Phase Begins

Dear participants,

we announce the start of the final Testing phase of our challenge. To participate in the final checkpoint two conditions must be met:

• you should fill out the registration form if you have not done so already. The registration form can be accessed via this link;
• you should submit your solution as a docker container and a report in TeX format via this form; you will find further guidelines inside. Your solutions are due 21.03.2025 23:59 UTC+0. You are only allowed one submission and cannot edit your answer.

Note the following:

• for this checkpoint, you are not supposed to submit any images; instead, we will run your solutions on the 200 RAW images which match the format of the data you used during the entire competition.
• Both a docker solution and a TeX report are required to be eligible for participation. Additionally, your solution is required to outperform a baseline solution we provided, evaluated by objective metrics, to be included in the final report.
• We ultimately reserve the right to reject a solution in case it does not follow the submission guidelines or fails to generate output properly.

We also impose an additional rule for evaluation regarding tie breaks. If two teams tie for a podium place (1^st - 3^rd), the winner of the tie will be the team with higher SSIM. Otherwise, teams will share a tied place.

The results will be available on the CodaLab page and on the official webpage of the challenge after evaluation.

If you have any additional questions, please let us know. We wish you the best of luck!

Validation 4 Results Released!

Dear participants,

We are excited to share the intermediate results of our competition based on the Validation 4 checkpoint.

For this evaluation, we considered all relevant solutions submitted before 14.03.2025, 00:00 UTC+0. A total of 11 submissions were received by the deadline.

Objective metric evaluation

We remind you that for objective metric evaluation, your final score is given by 0.4 * PSNRRank + 0.6 * SSIMRank. The standings are as follows:

Perceptual quality evaluation

We remind you that for the perceptual quality evaluation, we conduct a pairwise comparison of images generated by the solutions against the original DSLR images. Survey participants were asked to select which image in each pair better matched the original DSLR image. The score represents the average win percentage across all evaluated images. The standings are as follows:

Soon we will allow submissions for the final testing. Please be ready to submit your solutions as docker containers as well as reports describing your solution.

Thank you for participating and best of luck in the final checkpoint!

Validation 4 and Alternative Submission Options

Dear participants,

we are happy to anounce the start of the Validation 4 checkpoint. You can find the data at the Participate/Get Data section of the CodaLab page. Please submit your solutions until 13.03.2025 23:59 UTC+0.

Due to certain issues with CodaLab website and in preparation to the final Testing phase we provide an alternative way to submit your solution. You can fill out the form below and attach the resulting images as well as your docker solutions. Docker containers are not required for this checkpoint, however we encourage you to provide them as it will streamline the process of running your final solutions. Instructions can be found in the form.

Link to the submission form

Responses submitted through the form will be run manually. Due to that, participants are limited to two solutions per day. Please only employ this method of submission when submitting through CodaLab is impossible. The results will be maintained in a table on the CodaLab page. Please note that it will not be updated in real-time.

Solutions sent through both CodaLab and Google Forms will be eligible for participating in Validation 4 checkpoint MOS evaluation. The last solution (between CodaLab and Forms) will be used for evaluation.

We remind you that submitting your docker solutions as well as reports will be necessary for participating in the Testing phase.

We wish you the best of luck.

Google Forms Leaderboard (last upadated on 13.03.2025 17:30 UTC+0)

Validation 3 Results Released!

Dear participants,

We are excited to share the intermediate results of our competition based on the Validation 3 checkpoint.

For this evaluation, we considered all relevant solutions submitted before 05.03.2025, 00:00 UTC+0. A total of 8 submissions were received by the deadline. Participants who were unable to submit on time are encouraged to take part in the upcoming Validation 4 checkpoint.

Objective metric evaluation

We remind you that for objective metric evaluation, your final score is given by 0.4 * PSNRRank + 0.6 * SSIMRank. The standings are as follows:

Perceptual quality evaluation

For the perceptual quality evaluation, we conducted a pairwise comparison of images generated by the solutions against the original DSLR images. Participants were asked to select which image in each pair better matched the original DSLR image. The score represents the average win percentage across all evaluated images. The standings are as follows:

We remind you that the data for the next Validation 4 checkpoint will be released soon. Thank you for participating in our challenge!

MOS Delay and Additional Training Data

Dear participants,

we notify you that due to certain techical difficulties we have experienced with collecting MOS scores the results are delayed. We deeply apologize for this inconvenience. The results will be made available as soon as possible.

To accomodate this issue, the release and the deadline for the fourth validation checkpoint will be adjusted. The checkpoint data will be released on 11.03.2025 and the solutions will be due 13.03.2025. Please see the updated timeline.

To stimulate development of your solutions, we decided to release additional data. Camera images for the Validation 2 checkpoint are made available to you. Together with raw images from the Validation 2 checkpoint, they can be used as additional training data for your models. The download link in the "Get Data" section of the CodaLab page.

We will soon provide details on how to submit your final docker solutions and reports. Thank you for your interest in our challenge.

Additional Validation Checkpoint, Changes to the Timeline and Final Ranking Process Info

Dear participants,

we want to notify you about important changes coming to our challenge. Due to not being able to collect MOS during the Validation 2 checkpoint, we decided to add an additional Validation 4 checkpoint that will also include MOS calculation. That is so you can receive additional feedback on your solutions.

This also implies changes to the timeline. The testing phase will be postponed. Please see the updated timeline.

Finally we anounce how your final ranking in the objective metric leaderboard be evaluated. First, all participants wil be ranked based on their PSNR and SSIM scores; each participant will receive a PSNRRank and SSIMRank. Final score is calculated as Score = 0.4 * PSNRRank + 0.6 * SSIMRank, and participants with the lowest scores will be considered winners.

If you have any questions, please let us know.

Challenge overview

Welcome to the "Night Photography" challenge part of the NTIRE workshop at CVPR 2025. The challenge seeks to advance image processing methods for night photography by addressing the complexities of rendering images captured at night, particularly this year with raw mobile phone captures, and evaluating the results based on perceptual quality and computational efficiency.

The process of capturing and processing images taken by cameras relies on onboard processing to transform raw sensor data into polished photographs, typically encoded in a standard color space such as sRGB. Night photography, however, presents unique challenges not encountered in daylight scenes. Night images often feature complex lighting conditions, including multiple visible illuminants, and are characterized by distinctive noise patterns that make conventional photo-finishing techniques for daytime photography unsuitable. Additionally, widely used image quality metrics like SSIM and LPIPS are often ineffective for evaluating the nuances of night photography. In previous editions of this challenge, participants were tasked with processing raw night scene images into visually pleasing renders, assessed subjectively using mean opinion scores. These efforts have driven significant advancements in the field of night image processing.

This year's challenge introduces a new and fundamentally different approach while retaining the use of the mean opinion score as an evaluation metric. Moving away from purely subjective aesthetics, the 2025 challenge will adopt a potentially more objective framework by leveraging paired datasets captured with a Huawei smartphone and a high-end Sony camera using a beam splitter to ensure identical perspectives. Participants will be provided with raw Huawei smartphone images as input and corresponding processed Sony camera images as the expected output. The goal is to develop algorithms that transform the raw Huawei images—characterized by higher noise levels, vignetting, and reduced detail—into outputs that convincingly resemble the high-quality processed Sony images.

Mean opinion score will be employed to evaluate submissions, ensuring that human viewers, rather than automated metrics that might be exploited, determine how convincingly the processed Huawei images align with the processed Sony images. This focus on human perception underscores the importance of developing rendering algorithms that produce results that are not only technically accurate but also perceptually closer from the target images.

By addressing the unique limitations of mobile phone raw images and the inherent complexities of night photography, the 2025 challenge seeks to advance the state of the art in image processing. The combination of objective input-output alignment and perceptual validation through mean opinion scores provides a robust framework for fostering innovation in mobile-focused night photography rendering where objectivity has a more pronounced role, but is not expected to be abused in anyway due to the final subjective oversight.

Challenge Goal and Uniqueness

This challenge tackles the complexities of nighttime photography by leveraging paired datasets of raw Huawei smartphone images and processed Sony camera images, providing a clear ground truth for evaluation. The goal is to develop algorithms that process raw Huawei images to convincingly resemble the high-quality Sony outputs, addressing a long-standing challenge in computer vision.

Night photography is vital for applications like surveillance and security and also has artistic significance in creating stunning images. By combining objective ground-truth comparisons with human perception through mean opinion scores, the challenge ensures that results are both technically accurate and visually convincing.

This unique framework bridges mobile device constraints, low-light conditions, and human-centric evaluation, advancing the state of the art in night image processing.

 Reporting

In order to be eligible for the prizes, the participants will be required to send code and reports about their solutions in the form of short papers during the submission. If this report is not submitted, the participants that would otherwise win a prize will be passed over.

Prizes

There will be two prize categories:

1. Objective metrics (best score based on PSNR and SSIM)
2. Subjective metrics (best score based on MOS)

First three places from each category will be awarded 1000 USD, 650 USD, and 350 USD for the first, second, and third place, respectively.

Winners will receive a winner certificate and will have an opportunity to submit their paper to NTIRE'2025 and participate in the common report which also will be submitted to CVPR workshop.

Contacts and Questions

Please direct any and all questions at nightphotochallenge [at] gmail.com.