[doc][hackathon] To add AdamW Optimizer to the documentation #63252
Conversation
🔗 Helpful links
💊 CI failures summary and remediationsAs of commit 9671c27 (more details on the Dr. CI page):
1 failure not recognized by patterns:
This comment was automatically generated by Dr. CI (expand for details).Follow this link to opt-out of these comments for your Pull Requests.Please report bugs/suggestions to the (internal) Dr. CI Users group. |
iramazanli
added
docs-hackathon
hackathon
module: optimizer
iramazanli
force-pushed
the
adamw_algorithm_doc
branch
4 times, most recently
from
128aaea to
d985b28
Compare
iramazanli
force-pushed
the
adamw_algorithm_doc
branch
from
d985b28 to
fd9304f
Compare
Codecov Report
@@ Coverage Diff @@
## master #63252 +/- ##
==========================================
- Coverage 66.76% 66.65% -0.12%
==========================================
Files 710 710
Lines 92354 92395 +41
==========================================
- Hits 61658 61582 -76
- Misses 30696 30813 +117 |
iramazanli
force-pushed
the
adamw_algorithm_doc
branch
3 times, most recently
from
a711fbe to
409c1e7
Compare
torch/optim/adamw.py
Outdated
There was a problem hiding this comment.
Is that correct? We don't actually do weight decay this way for this one no?
There was a problem hiding this comment.
yes, you're completely right !
torch/optim/adamw.py
Outdated
There was a problem hiding this comment.
Where is the \lambda \theta_{t-1} coming from?
There was a problem hiding this comment.
i think it shouldn't exist, you're right again !
iramazanli
force-pushed
the
adamw_algorithm_doc
branch
5 times, most recently
from
45cdbf8 to
7b1b7ea
Compare
torch/optim/adamw.py
Outdated
There was a problem hiding this comment.
This is missing a multiplication by \gamma no?
There was a problem hiding this comment.
yes, i added it. thanks for pointing it out
iramazanli
force-pushed
the
adamw_algorithm_doc
branch
from
7b1b7ea to
9671c27
Compare
|
@iramazanli has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
|
@iramazanli merged this pull request in 5b21f17. |
3 participants
It has been discussed before that adding description of Optimization algorithms to PyTorch Core documentation may result in a nice Optimization research tutorial. In the following tracking issue we mentioned about all the necessary algorithms and links to the originally published paper #63236.
In this PR we are adding description of AdamW Algorithm to the documentation. For more details, we refer to the paper here https://arxiv.org/abs/1711.05101
cc @vincentqb @iramazanli