[c10d][logging] Add wait counter for time spent in object to tensor and tensor to object #140414
Conversation
…nd tensor to object [ghstack-poisoned]
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/140414
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 0f43adf with merge base f77eb07 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
pytorch-bot
bot
added
oncall: distributed
fduwjj
added
ciflow/trunk
|
@wconstab good question, it is already used in TorchCompile already so the risk is kind of low. |
|
I worked with Boris to verify that on a internal typical workload, the perf difference after adding this counter is less than 5% of latency change. So we got green light for a merge. |
|
@wz337 let's see how this goes. The init time is just one time, so it's not as expensive as this one.. |
|
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
…nd tensor to object (pytorch#140414) Originally we want to leverage the timer logger to measure the time spent in object to tensor and tensor to object (pytorch#139757) But it gets reverted (internally) because of a performance regression. We now use wait counter instead which is more lightweight. Pull Request resolved: pytorch#140414 Approved by: https://github.com/c-p-i-o, https://github.com/XilunWu, https://github.com/wz337
6 participants
Stack from ghstack (oldest at bottom):
Originally we want to leverage the timer logger to measure the time spent in object to tensor and tensor to object (#139757) But it gets reverted (internally) because of a performance regression. We now use wait counter instead which is more lightweight.
cc @H-Huang @awgu @kwen2501 @wanchaol @fegin @wz337 @wconstab @d4l3k @c-p-i-o