This issue is for tracking documentation of Optimization Algorithms.

Currently PyTorch provides documentation for Optimizers by referring to papers they are provided. It could be nice if we represent algorithms within PyTorch documentation itself, so our documentation would be a good optimization research tutorial as well

• Adam : https://arxiv.org/pdf/1412.6980.pdf
• Adadelta : https://arxiv.org/abs/1212.5701
• Adagrad : https://jmlr.org/papers/v12/duchi11a.html
• AdamW : https://arxiv.org/abs/1711.05101
• Adamax : https://arxiv.org/abs/1412.6980
• ASGD : https://dl.acm.org/doi/10.1137/0330046
• LBFGS https://www.cs.ubc.ca/~schmidtm/Software/minFunc.html
• RMSProp : https://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf
• RProp
• SGD : http://www.cs.toronto.edu/~hinton/absps/momentum.pdf

cc @vincentqb @iramazanli