This was a typo in the previous doc right?

Yes, actually \alpha generally stands for learning rate, however for the specific case of RMSProp alpha stands for coefficient of square average in the literature (including in PyTorch parameter alpha - smoothing constant).

So, the learning rate scheduler needed to have some other representation which i selected \gamma. We can change it to some other character if you like :)

I don't have a specific preference. If there is one character that is not used by any of the optimizers, we should use that one so that we can make all the optimizers consistent.

Cool, so i have changed everything to \gamma for learning rate :) now, there should be consistency.

What about the momentum, weight decay and centered arguments?

at this point I provided algorithms as they are provided in their corresponding research papers. For example Adam has the following look

So I tried to keep consistent with papers. Let me know if you think that we should go for different design.

I think we can link to the paper if people want to see the original algorithm. But the one we provide should properly reflect what is actually implemented. And all the arguments we support.

This should be f_{t-1} no?

actually in many papers the notation is "f_t" in particular "f_t (\theta_{t-1})"

For example

• in the original paper of Adam in the page 2, in Algorithm 1 https://arxiv.org/pdf/1412.6980.pdf
• in the paper of RAdam page 5, Algorithm 2 : https://arxiv.org/pdf/1412.6980.pdf

To my understanding the idea is we use parameter \theta_{t-1} because the latest computed parameter is at timestep t-1. However, the stochastic gradient is computed at this step -t, that's why notation is f_t instead of f_{t-1}

Ho that's interesting.. But as long as it is consistent for all optimizers, I'm happy :)

it is consistent in PyTorch documentation and papers :)

g^{ave} is missing from here.

oh, thanks for pointing out :) and its fixed

The momentum buffer is initialized with 0s in the code. Why is this one 1s?

yes , you're right, i modified it