[c10d][fr] wait counter for dump function #140823
Conversation
pytorch-bot
bot
added
oncall: distributed
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/140823
Note: Links to docs will display an error until the docs builds have been completed. ❗ 1 Active SEVsThere are 1 currently active SEVs. If your PR is affected, please view them below: ✅ No FailuresAs of commit 520f229 with merge base 9602f56 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
|
This pull request was exported from Phabricator. Differential Revision: D65823433 |
c-p-i-o
force-pushed
the
export-D65823433
branch
from
de6df66 to
74ea816
Compare
|
This pull request was exported from Phabricator. Differential Revision: D65823433 |
Summary: Add a wait counter for the dump function. This is useful to see if we get stuck in the dump function and never return for a particular job. Test Plan: Tested locally I and see `pytorch.wait_counter.NCCLTraceBuffer__dump.busy_time_us.sum.60` in ODS. Reviewed By: fduwjj Differential Revision: D65823433
c-p-i-o
force-pushed
the
export-D65823433
branch
from
74ea816 to
520f229
Compare
|
This pull request was exported from Phabricator. Differential Revision: D65823433 |
|
@pytorchbot merge (Initiating merge automatically since Phabricator Diff has merged) |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Summary: Add a wait counter for the dump function. This is useful to see if we get stuck in the dump function and never return for a particular job. Test Plan: Tested locally I and see `pytorch.wait_counter.NCCLTraceBuffer__dump.busy_time_us.sum.60` in ODS. Differential Revision: D65823433 Pull Request resolved: pytorch#140823 Approved by: https://github.com/fduwjj
4 participants
Summary:
Add a wait counter for the dump function.
This is useful to see if we get stuck in the dump function and never return for a particular job.
Test Plan: Tested locally I and see
pytorch.wait_counter.NCCLTraceBuffer__dump.busy_time_us.sum.60in ODS.Differential Revision: D65823433
cc @H-Huang @awgu @kwen2501 @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k