[C10D] Support group_dst/group_src in c10d send/recv object_list #140847
Conversation
[ghstack-poisoned]
pytorch-bot
bot
added
oncall: distributed
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/140847
Note: Links to docs will display an error until the docs builds have been completed. ❗ 1 Active SEVsThere are 1 currently active SEVs. If your PR is affected, please view them below: ✅ No FailuresAs of commit 440d2cf with merge base b379a28 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
…t_list" [ghstack-poisoned]
…t_list" [ghstack-poisoned]
…t_list" [ghstack-poisoned]
…t_list" [ghstack-poisoned]
…t_list" [ghstack-poisoned]
…t_list" [ghstack-poisoned]
wconstab
requested review from
H-Huang and
kwen2501
and removed request for
kwen2501
…t_list" Also add mypy annotations Partially addresses RFC 0042 (pytorch/rfcs#71) See more details/motivation in #140460 [ghstack-poisoned]
| serialized and converted to tensors which are moved to the | ||
| ``device`` before sending. Default is ``None``. | ||
| group_dst (int, optional): Destination rank on ``group``. |
There was a problem hiding this comment.
if you specify group_dst is there a requirement to also specify group?
There was a problem hiding this comment.
no. Technically, group defaults to 'default_group' and that's documented. So I allowed using group_dst along with default group. (This means in that case group_dst would be == global_dst).
| x = [{}] | ||
| c10d.recv_object_list(x, src=self.rank + 1, group=subgroup, device=device) | ||
| if group_rank: | ||
| c10d.recv_object_list(x, group_src=1, group=subgroup, device=device) |
There was a problem hiding this comment.
instead of group_src=1 is there a way to do something like subgroup.local_rank() + 1?
There was a problem hiding this comment.
yea, it would be equivalent. I just left it this way bc the test already hardcodes 4 ranks and 2 subgroups, so this is pretty clear IMO
…t_list" Also add mypy annotations Partially addresses RFC 0042 (pytorch/rfcs#71) See more details/motivation in #140460 [ghstack-poisoned]
|
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
…orch#140847) Also add mypy annotations Partially addresses RFC 0042 (pytorch/rfcs#71) See more details/motivation in pytorch#140460 Pull Request resolved: pytorch#140847 Approved by: https://github.com/H-Huang ghstack dependencies: pytorch#140843
ghstack-source-id: 1fdcdbd Pull Request resolved: pytorch/pytorch#140847
3 participants
Stack from ghstack (oldest at bottom):
Also add mypy annotations
Partially addresses RFC 0042 (pytorch/rfcs#71)
See more details/motivation in #140460
cc @H-Huang @awgu @kwen2501 @wanchaol @fegin @fduwjj @wz337 @d4l3k @c-p-i-o