I got some questions and thought is better to ask them all together:
1. Can we zip other files than the agent.py and meta-data ( weights-pretrained models)?
2. Is it possible to get the whole learning curve till the queried budget? So far the agent only returns the performance that is closer but less than the queried budget.
3. Can the agent save files to disk while meta-training?
4. Why is the scoring going downwards sometimes for some datasets? I can see that for some LC plots from the current submissions, where the learning curve is not monotonically increasing.
5. Can an existing user join a new team and not use the old account?
Best,
Pineda
Additionally:
Yesterday and today I am getting this output error when I submit a solution:
"
Traceback (most recent call last):
File "/tmp/codalab/tmpG1QVo5/run/program/score.py", line 368, in <module>
df = pd.read_csv(output_file, names=['algo_index', 'algo_time_spent', 'total_time_spent', 'score'])
File "/opt/conda/lib/python3.7/site-packages/pandas/io/parsers.py", line 702, in parser_f
return _read(filepath_or_buffer, kwds)
File "/opt/conda/lib/python3.7/site-packages/pandas/io/parsers.py", line 429, in _read
parser = TextFileReader(filepath_or_buffer, **kwds)
File "/opt/conda/lib/python3.7/site-packages/pandas/io/parsers.py", line 895, in __init__
self._make_engine(self.engine)
File "/opt/conda/lib/python3.7/site-packages/pandas/io/parsers.py", line 1122, in _make_engine
self._engine = CParserWrapper(self.f, **self.options)
File "/opt/conda/lib/python3.7/site-packages/pandas/io/parsers.py", line 1853, in __init__
self._reader = parsers.TextReader(src, **kwds)
File "pandas/_libs/parsers.pyx", line 387, in pandas._libs.parsers.TextReader.__cinit__
File "pandas/_libs/parsers.pyx", line 705, in pandas._libs.parsers.TextReader._setup_parser_source
FileNotFoundError: [Errno 2] File b'/tmp/codalab/tmpG1QVo5/run/input/res/adult.csv' does not exist: b'/tmp/codalab/tmpG1QVo5/run/input/res/adult.csv'
"
Posted by: pineda @ Jan. 31, 2022, 7:19 p.m.Hi,
1. Can we zip other files than the agent.py and meta-data ( weights-pretrained models)?
YES, you can.
2. Is it possible to get the whole learning curve till the queried budget? So far the agent only returns the performance that is closer but less than the queried budget.
In case there is no performance score pre-recorded at the queried budget, the latest score obtained will be returned together with the actual time budget (which may be less than the queried budget) needed to achieve that score. However, the agent must pay the full queried budget to encourage learning smarter time management policy. If it wants to achieve a new score, it needs to pay a sufficient amount of time budget.
3. Can the agent save files to disk while meta-training?
NO, participants are not allowed to write anything to disk.
4. Why is the scoring going downwards sometimes for some datasets? I can see that for some LC plots from the current submissions, where the learning curve is not monotonically increasing.
The learning curve plots are the agents’ learning curves. They’re not necessarily increasing monotonically. If an agent explores worse algorithms, its learning curve may go downward.
5. Can an existing user join a new team and not use the old account?
Yes, you can. In that case, we can manually revoke your old account from our competition.
Regarding your error, we checked and it seemed that you forgot to zip the metadata file together with your agent (please check again our instructions in the README file).
Best regards,
Hung