To add Rectified Adam Description to Documentation #63772

iramazanli · 2021-08-23T16:22:34Z

It has been discussed before that adding description of Optimization algorithms to PyTorch Core documentation may result in a nice Optimization research tutorial. In the following tracking issue we mentioned about all the necessary algorithms and links to the originally published paper #63236.

In this PR we are adding description of Rectified Adam Algorithm to the documentation. For more details, we refer to the paper https://arxiv.org/abs/1908.03265

facebook-github-bot · 2021-08-23T16:22:40Z

🔗 Helpful links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/63772

💊 CI failures summary and remediations

As of commit bddbd8d (more details on the Dr. CI page):

3/3 failures introduced in this PR

🕵️ 3 new failures recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

win-vs2019-cpu-py3 / build (1/3)

Step: "Build" (full log | diagnosis details | 🔁 rerun)

2021-09-08T21:40:10.1605021Z ls: cannot access ...d/win_tmp/ci_scripts/*': No such file or directory

2021-09-08T21:40:09.9430089Z ++ cygpath -w /c/actions-runner/_work/pytorch/pytorch/pytorch-1215013221/build/win_tmp
2021-09-08T21:40:10.0032237Z + TMP_DIR_WIN='C:\actions-runner\_work\pytorch\pytorch\pytorch-1215013221\build\win_tmp'
2021-09-08T21:40:10.0032853Z + export TMP_DIR_WIN
2021-09-08T21:40:10.0033294Z + export PYTORCH_FINAL_PACKAGE_DIR=/c/1215013221/build-results/
2021-09-08T21:40:10.0033827Z + PYTORCH_FINAL_PACKAGE_DIR=/c/1215013221/build-results/
2021-09-08T21:40:10.0034266Z + [[ -n /c/1215013221/build-results/ ]]
2021-09-08T21:40:10.0034638Z + mkdir -p /c/1215013221/build-results/
2021-09-08T21:40:10.1106693Z + CI_SCRIPTS_DIR=/c/actions-runner/_work/pytorch/pytorch/pytorch-1215013221/build/win_tmp/ci_scripts
2021-09-08T21:40:10.1107552Z + mkdir -p /c/actions-runner/_work/pytorch/pytorch/pytorch-1215013221/build/win_tmp/ci_scripts
2021-09-08T21:40:10.1293138Z ++ ls '/c/actions-runner/_work/pytorch/pytorch/pytorch-1215013221/build/win_tmp/ci_scripts/*'
2021-09-08T21:40:10.1605021Z ls: cannot access '/c/actions-runner/_work/pytorch/pytorch/pytorch-1215013221/build/win_tmp/ci_scripts/*': No such file or directory
2021-09-08T21:40:10.1607908Z + '[' -n '' ']'
2021-09-08T21:40:10.1608667Z + export SCRIPT_HELPERS_DIR=/c/actions-runner/_work/pytorch/pytorch/pytorch-1215013221/.jenkins/pytorch/win-test-helpers
2021-09-08T21:40:10.1609719Z + SCRIPT_HELPERS_DIR=/c/actions-runner/_work/pytorch/pytorch/pytorch-1215013221/.jenkins/pytorch/win-test-helpers
2021-09-08T21:40:10.1610305Z + set +ex
2021-09-08T21:40:18.2153149Z + /c/actions-runner/_work/pytorch/pytorch/pytorch-1215013221/.jenkins/pytorch/win-test-helpers/build_pytorch.bat
2021-09-08T21:40:18.2501464Z 
2021-09-08T21:40:18.2502417Z C:\actions-runner\_work\pytorch\pytorch\pytorch-1215013221>if "" == "1" (set BUILD_TYPE=debug )  ELSE (set BUILD_TYPE=release ) 
2021-09-08T21:40:18.2505628Z 
2021-09-08T21:40:18.2507858Z C:\actions-runner\_work\pytorch\pytorch\pytorch-1215013221>set PATH=C:\Program Files\CMake\bin;C:\Program Files\7-Zip;C:\ProgramData\chocolatey\bin;C:\Program Files\Git\cmd;C:\Program Files\Amazon\AWSCLI;C:\Program Files\Amazon\AWSCLI\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0;C:\Windows\System32\OpenSSH;C:\Program Files\Amazon\cfn-bootstrap;C:\ProgramData\chocolatey\bin;C:\Program Files\Amazon\AWSCLIV2;C:\Program Files\Git\cmd;C:\Program Files\Git\mingw64\bin;C:\Program Files\Git\usr\bin;C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit;C:\Users\runneruser\AppData\Local\Microsoft\WindowsApps 
2021-09-08T21:40:18.2509995Z

pytorch_xla_linux_bionic_py3_6_clang9_build (2/3)

Step: "(Optional) Merge target branch" (full log | diagnosis details | 🔁 rerun)

Automatic merge failed; fix conflicts and then commit the result.

CONFLICT (add/add): Merge conflict in .circleci/docker/build.sh
Auto-merging .circleci/docker/build.sh
CONFLICT (add/add): Merge conflict in .circleci/config.yml
Auto-merging .circleci/config.yml
CONFLICT (add/add): Merge conflict in .circleci/cimodel/data/pytorch_build_definitions.py
Auto-merging .circleci/cimodel/data/pytorch_build_definitions.py
CONFLICT (add/add): Merge conflict in .circleci/cimodel/data/pytorch_build_data.py
Auto-merging .circleci/cimodel/data/pytorch_build_data.py
CONFLICT (add/add): Merge conflict in .bazelrc
Auto-merging .bazelrc
Automatic merge failed; fix conflicts and then commit the result.


Exited with code exit status 1

pytorch_linux_xenial_py3_6_gcc5_4_build (3/3)

Step: "(Optional) Merge target branch" (full log | diagnosis details | 🔁 rerun)

Automatic merge failed; fix conflicts and then commit the result.

CONFLICT (add/add): Merge conflict in .circleci/docker/build.sh
Auto-merging .circleci/docker/build.sh
CONFLICT (add/add): Merge conflict in .circleci/config.yml
Auto-merging .circleci/config.yml
CONFLICT (add/add): Merge conflict in .circleci/cimodel/data/pytorch_build_definitions.py
Auto-merging .circleci/cimodel/data/pytorch_build_definitions.py
CONFLICT (add/add): Merge conflict in .circleci/cimodel/data/pytorch_build_data.py
Auto-merging .circleci/cimodel/data/pytorch_build_data.py
CONFLICT (add/add): Merge conflict in .bazelrc
Auto-merging .bazelrc
Automatic merge failed; fix conflicts and then commit the result.


Exited with code exit status 1

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

codecov · 2021-08-24T22:57:41Z

Codecov Report

Merging #63772 (4f47ee1) into master (f0d2742) will decrease coverage by 0.01%.
The diff coverage is n/a.

❗ Current head 4f47ee1 differs from pull request most recent head b1e9dd8. Consider uploading reports for the commit b1e9dd8 to get more accurate results

@@            Coverage Diff             @@
##           master   #63772      +/-   ##
==========================================
- Coverage   67.09%   67.07%   -0.02%     
==========================================
  Files         692      691       -1     
  Lines       90579    90571       -8     
==========================================
- Hits        60774    60754      -20     
- Misses      29805    29817      +12

albanD · 2021-09-07T16:38:07Z

torch/optim/radam.py

It should be v_t + eps here no?

facebook-github-bot · 2021-09-09T12:47:55Z

@iramazanli has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot · 2021-09-09T14:12:48Z

@iramazanli merged this pull request in 239366c.

facebook-github-bot added the cla signed label Aug 23, 2021

iramazanli force-pushed the radam_algorithm_doc branch 4 times, most recently from adc1d4b to 4f47ee1 Compare August 24, 2021 19:11

iramazanli force-pushed the radam_algorithm_doc branch 2 times, most recently from 4cbd21c to b1e9dd8 Compare August 26, 2021 23:01

iramazanli changed the title ~~To add Rectified Adam to PyTorch Optim~~ To add Rectified Adam Description to Documentation Aug 27, 2021

iramazanli force-pushed the radam_algorithm_doc branch 3 times, most recently from 03e7475 to 383515d Compare August 27, 2021 19:12

iramazanli requested a review from albanD August 27, 2021 19:34

iramazanli force-pushed the radam_algorithm_doc branch 3 times, most recently from ef83c9d to b33dadb Compare August 27, 2021 20:43

albanD reviewed Sep 7, 2021

View reviewed changes

torch/optim/radam.py Outdated

Copy link

Collaborator

albanD Sep 7, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should be v_t + eps here no?

To add Rectified Adam to PyTorch Optim

bddbd8d

iramazanli force-pushed the radam_algorithm_doc branch from b33dadb to bddbd8d Compare September 8, 2021 21:32

albanD approved these changes Sep 8, 2021

View reviewed changes

facebook-github-bot closed this in 239366c Sep 9, 2021

facebook-github-bot added the Merged label Sep 9, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

To add Rectified Adam Description to Documentation #63772

To add Rectified Adam Description to Documentation #63772

iramazanli commented Aug 23, 2021 •

edited

Loading

Uh oh!

facebook-github-bot commented Aug 23, 2021 •

edited

Loading

Uh oh!

codecov bot commented Aug 24, 2021 •

edited

Loading

Uh oh!

albanD Sep 7, 2021

Uh oh!

facebook-github-bot commented Sep 9, 2021

Uh oh!

facebook-github-bot commented Sep 9, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

To add Rectified Adam Description to Documentation #63772

To add Rectified Adam Description to Documentation #63772

Conversation

iramazanli commented Aug 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

facebook-github-bot commented Aug 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful links

💊 CI failures summary and remediations

🕵️ 3 new failures recognized by patterns

win-vs2019-cpu-py3 / build (1/3)

pytorch_xla_linux_bionic_py3_6_clang9_build (2/3)

pytorch_linux_xenial_py3_6_gcc5_4_build (3/3)

Uh oh!

codecov bot commented Aug 24, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

albanD Sep 7, 2021

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot commented Sep 9, 2021

Uh oh!

facebook-github-bot commented Sep 9, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

iramazanli commented Aug 23, 2021 •

edited

Loading

facebook-github-bot commented Aug 23, 2021 •

edited

Loading

codecov bot commented Aug 24, 2021 •

edited

Loading