Hi,
I have some questions about complexity measurements:
1. Are the final measurements exactly the same with the provided Python scoring scripts?
2. There may be some processings not contained in the torch model, such as mapping LDR inputs to HDR domain in the baseline method. How are these operations considered in the evaluation?
Thanks.
Hello buzzli,
With respect to 1., the PSNR scores will be obtained using exactly the same implementations provided in the python scoring scripts. For this track, the ranking will be based on runtime and computational complexity (in terms of number of operations). Participants should always aim at reducing the runtime and number of operations in their solutions. We provide examples on how to measure both metrics so that participants can measure them while developing their solutions, but we will ultimately carefully inspect and double check the submitted solutions to ensure runtime measurements are hardware consistent across all teams, and also to ensure that the total number of operations is correct.
With respect to 2., all pre-processing should be included in the the torch model and counted towards runtime and number of operations. We have updated the example script to reflect this more clearly.
Best regards,
Edu.
Hi Edu,
Thank you for your reply. I still need some help about this issue.
I found exposure values and the steps are not fixed in training and validation samples. For example, there are [-3, 0, 3] and [-2, ,0, 2]. I suppose that means our methods should read exposure values on the fly to normalize each sample.
However, In the evaluation example, model inputs are suggested to be a singe tensor with given shape. So I'm not sure how to pass the expo values to do the normalization.
Is there any suggestion about this?
Hello buzzli,
All the pre-processing should be included in the computation of number of operations and in the runtime measurements. You are free to utilize any pre-processing you want or none at all, but in general one data point is fed to the model and the challenge data has a constant size, so it is unlikely that participants need to modify that. In practise, image-based processing is what generally dominates the number of operations, and things like the exposure value normalization will have an almost negligible impact for most of the cases. We have recently updated the related scripts to further clarify its use.
We would like to outline that the scripts we provide are for guidance only, and are just an illustration of how to perform the measurements for a "toy model" and "toy processing", but they are by no means a rigid template that will fit all possible designs, nor prescribe any best practices for how to process the data or create a neural network.
There is no problem if you want fixed value exposures instead of floating exposure, as I said this is just a "toy example" and not a prescription of what participants should do. Please just modify the code accordingly so that it reflects all the computations happening within your model. Generally, the size of the input data remains unchanged and participants just need to ensure that any resizing or similar processing is accounted in the measurements.
Best regards,
Edu.