I am reducing the learning rate with ReduceLROnPlateau during training every 10 epochs so when it is reduced the weights could be far from the best ones. Does it make sense to restore the best weights before reducing the learning rate? Or is it usually better to continue with the current weights?
Posted by: axelitama @ Nov. 9, 2022, 9:26 p.m.> I am reducing the learning rate with ReduceLROnPlateau during training every 10 epochs so when it is reduced the weights could be far from the best ones. Does it make sense to restore the best weights before reducing the learning rate? Or is it usually better to continue with the current weights?
I think the objective of the learning rate reduction only makes sense on the plateau: so if you reduce your best weights, those might not be a plateau, causing only a very slow convergence rate (which may get your model stopped by the EarlyStop if it becomes too slow).
Am I making sense?