Dear Organizers,
In Evaluation, it is mentioned that "models should have a small computational load/complexity, below 50 MACs (multiply-accumulate operations)." There is some uncertainty regarding this specification, as it seems that 50 MACs represents an extremely small computational complexity. Could there possibly be a missing unit for the MACs?
Best regards
Posted by: Caesar_D @ June 13, 2024, 4:29 a.m.Dear Participant(s),
Thank you very much for the question.
The 50 MACs were decided as assumed to run MobileNetV2 for a 4k image without any sampling. We do not necessarily expect to have a model submission more compact than MobileNetV2 but expect participants to reduce the computations by costume methods such as quantizing, sampling the input data (e.g., taking a small number of patches from a 4k image or resizing the image) or any other novel methods (e.g., model pruning).
Best regards,
The organizers
Dear Organizers,
Thank you for the clarification. I understand that running MobileNetV2 for a 4K image requires approximately 54G FLOPs, which is equivalent to around 27G MACs considering the 1:2 FLOPs to MACs ratio.
Given the initial guideline of "below 50 MACs," could you confirm if the correct computational complexity threshold we should aim for is "below 50G MACs"? This would help in aligning our model optimization efforts accurately.
Best regards,
clearlon
Dear Participant,
Thank you for the question.
For the reference to our MAC measurement, please refer to the GitHub repo below; if you see any mistake in the MAC measurement, please report that to us.
https://github.com/mv-lab/VideoAI-Speedrun
Best regards,
Saman
Dear Organizers,
I am currently working on calculating the MACs for running MobileNetV2 on a 4K image without any sampling.
Despite using the examples provided in https://github.com/mv-lab/VideoAI-Speedrun, I am getting a result of 20.32G MACs, which seems incorrect.
Below is the code I used:
‘’‘
import torch
from ptflops import get_model_complexity_info
import timm
with torch.cuda.device(0):
net = timm.create_model('mobilenetv2_035', pretrained=False)
macs, params = get_model_complexity_info(net, (3, 4096, 4096), as_strings=True, backend='pytorch', print_per_layer_stat=True, verbose=True)
print('{:<30} {:<8}'.format('Computational complexity: ', macs))
print('{:<30} {:<8}'.format('Number of parameters: ', params))
’‘’
Could you please review the code or provide your code for MACs calculation?
Best regards,
Thanks :)
Mike
Posted by: Ohayou2024 @ July 1, 2024, 9:51 a.m.Dear Mike,
I have reviewed your code and it appears to be correct. However, I have identified two differences between your example and the measurement we use in our calculations.
Firstly, you are using a different version of MobileNet; we used mobilenetv2_100. Secondly, please ensure that the resolution is set to 4K (2160x3840) for the calculations. Although this resolution is not for a square shape, it serves as the basis for our measurements.
Below is the revised version of your code to align with our measurement standards:
import torch
from ptflops import get_model_complexity_info
import timm
with torch.cuda.device(0):
net = timm.create_model('mobilenetv2_100', pretrained=False)
macs, params = get_model_complexity_info(net, (3, 2160, 3840), as_strings=True, backend='pytorch', print_per_layer_stat=True, verbose=True)
print('{:<30} {:<8}'.format('Computational complexity: ', macs))
print('{:<30} {:<8}'.format('Number of parameters: ', params))
Best regards,
Saman
Dear organizers,
I would like to inquire whether the model's MAC size influences the competition ranking, or if the final ranking solely depends on the five indicators, like SROCC, MAE etc.
Meanwhile, is there a constraint on inference time. For example, I have designed a model that supports 224*224 resolution (MAC less than 50G), but it needs to crop about 50 times for 4k to get the final quality score. This leads to an increase in inference time even though the MAC of the model is small, may I ask if this situation is in accordance with the rules.
Best regards,
Mike
[made a slight revision in evaluation criteria]
Hi Mike,
Thanks for the question.
In general, MAC measurement only impacts whether your model is considered for competition or not. Your model will not be ranked based on your Mac measurement (if you pass the criteria). For example, suppose your model has 20 GMac for a 4K image, and another model has 40 GMac for the same image. In that case, your model will not have any advantage (both models will be compared based on competition criteria, which are your rank in correlation metrics (SROCC, PLCC, KRCC) and your rank in error metrics (MAE + RMSE). Please note that the idea is to develop a model for mobile devices.
Regarding your question about MAC measurement, we only measure the MACs based on the MACs you require to predict the quality of a 4K image (3480x2160x3). Here are some examples:
- If you pass a 4K (3480x2160x3) image without any cropping to "mobilenetv2_100" you will get 50.64 GMac which pass our criteria.
- If you resize the image from 4k to 224x224x3 you will get 307.01 MMac which leads to far lower Mac you require (for "mobilenetv2_100").
- If you take 10 patches/crops of 224x224x3 from 4k images and pass through "mobilenetv2_100" then you will get around 3.190 GMac.
- If you take 1000 patches/crops of 224x224x3 from 4k images and pass through "mobilenetv2_100," then you will get around 319.08 GMac, which you don't pass the criteria.
In short what is important is how many patches/crops (an size of it) you need to pass to your model and the Mac must be measured based on that. The time (and process) you need to crop the images is not needed to be considered in the complexity measurment.
We don't measure the inference time because we need to fix the GPU and CPU model, which makes it difficult to measure at your end if you don't have the same GPU and CPU model we test. Mac's measurement does not have this issue.
Last point: even if your model does not pass the criteria, I encourage you to submit your model for the competition, as we have an extra prize for innovative ideas in the model development.
Best regards,
Saman
Hi, I'd like to know if the model calculations are strictly at <=50GMACs, or are they allowed to go a little beyond that?
Posted by: Czw @ July 11, 2024, 3:33 a.m.Hello, I want to know the details about MAC registration.
Given that my model's MACs are less than 50G when using UHD Image, is it allowed to use techniques (e.g. TTA) that increase computation at the inference stage? (Assuming that if i use the technique, it exceeds 50G MACs)
Dear Participant,
Concerning your question about the strict 50 GMac limit, we request that you adhere to this threshold. If your model slightly exceeds it, you might consider slight center cropping for the competition. However, if you are also submitting a paper, feel free to report based on your primary model.
For your second question, our GMac measurement is based on inference. This means that if you need to pass each test image through your model multiple times, the GMac will accumulate accordingly.
Best regards,
Saman