Add wait counter for nccl abort #136067
Add wait counter for nccl abort #136067
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/136067
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 75445cc with merge base 48d18fb ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
pytorch-bot
bot
added
oncall: distributed
|
This pull request was exported from Phabricator. Differential Revision: D62675010 |
|
@atuljangra do you mind adding a brief comment above the code describing what the code does? |
|
@kwen2501 Added some more details int eh description. |
|
@atuljangra Thanks. Maybe I confused you :) What I meant was to copy the PR description into the code as a comment (so that readers know what that line of code is for). |
|
@kwen2501 Ah got it, will update. |
atuljangra
force-pushed
the
export-D62675010
branch
from
1717e5d to
fa00385
Compare
|
This pull request was exported from Phabricator. Differential Revision: D62675010 |
atuljangra
force-pushed
the
export-D62675010
branch
from
fa00385 to
f660774
Compare
|
This pull request was exported from Phabricator. Differential Revision: D62675010 |
atuljangra
force-pushed
the
export-D62675010
branch
from
f660774 to
4d5f6c6
Compare
|
This pull request was exported from Phabricator. Differential Revision: D62675010 |
atuljangra
force-pushed
the
export-D62675010
branch
from
4d5f6c6 to
1fb266f
Compare
|
This pull request was exported from Phabricator. Differential Revision: D62675010 |
Summary: Pull Request resolved: pytorch#136067 Quite a few times, we see the NCCL PG abort taking too long. There's no easy way to measure this, so let's add a counter to measure this across the stack. Example job where this would have been useful: f641288746-TrainingApplication Test Plan: Unit tests https://www.internalfb.com/mlhub/pipelines/runs/mast/aps-atuljangra-8519d57d57?job_attempt=0&env=PRODUCTION Reviewed By: c-p-i-o Differential Revision: D62675010
atuljangra
force-pushed
the
export-D62675010
branch
from
1fb266f to
75445cc
Compare
|
This pull request was exported from Phabricator. Differential Revision: D62675010 |
|
@pytorchbot merge -f 'Landed internally' (Initiating merge automatically since Phabricator Diff has merged, using force because this PR might not pass merge_rules.json but landed internally) |
Merge startedYour change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Summary: Quite a few times, we see the NCCL PG abort taking too long. There's no easy way to measure this, so let's add a counter to measure this across the stack. This will help us measure how much time we take the NCCL abort. Test Plan: Unit tests Reviewed By: c-p-i-o Differential Revision: D62675010 Pull Request resolved: pytorch#136067 Approved by: https://github.com/fduwjj
5 participants
Summary:
Quite a few times, we see the NCCL PG abort taking too long. There's no easy way to measure this, so let's add a counter to measure this across the stack.
This will help us measure how much time we take the NCCL abort.
Test Plan:
Unit tests
Reviewed By: c-p-i-o
Differential Revision: D62675010
cc @XilunWu @H-Huang @awgu @kwen2501 @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k @c-p-i-o