[C10d][Gloo] Enable complex datatype support in ProcessGroupGloo #156633

shunzhiwen · 2025-06-23T20:09:30Z

Enable communication of tensors with Complex datatype in ProcessGroupGloo, similar to how ProcessGroupNCCL handles it.
Move a function, which checks if Complex datatype is supported by a reduce operation, from ProcessGroupNCCL.cpp into a new file to be shared with ProcessGroupGloo.

cc @H-Huang @awgu @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k @pragupta @ezyang @msaroufim @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @chenyang78 @kadeng @muchulee8 @amjames @chauhang @aakhundov @coconutruben

pytorch-bot · 2025-06-23T20:09:34Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/156633

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 5b2c992 with merge base 0e45023 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

linux-foundation-easycla · 2025-06-23T20:09:35Z

The committers listed above are authorized under a signed CLA.

✅ login: shunzhiwen / name: Shunzhi Wen (5b2c992, adc6f1b)

shunzhiwen · 2025-06-30T18:44:19Z

Renamed function to avoid conflicts with third party torch-xpu-ops. It defines a function with the same name under the same namespace in ProcessGroupXCCL.cpp.

d4l3k

Code changes look very reasonable to me -- can we add some unit tests in test_c10d_gloo to check for complex type support?

shunzhiwen · 2025-08-18T19:30:12Z

Added a few test cases. Moved FFTModel in test_c10d_nccl.py into test_c10d_common.py to reuse here.

d4l3k

LGTM

d4l3k · 2025-08-19T19:14:29Z

test/distributed/test_c10d_common.py

        return self.conv3(x)


+class FFTModel(nn.Module):


can you add a comment about what this is supposed to test? it's a bit unclear just by itself

d4l3k · 2025-08-19T19:15:53Z

test/distributed/test_c10d_gloo.py

+        out = ddp_model(inp)
+        loss = torch.sum(out)
+        loss.backward()
+        optimizer.step()


would also be nice to add explicit tests for each of the added operations -- I think DDP only covers allreduce and broadcast?

See _test_broadcast_basics for how to make those calls directly

More individual test cases are added.

d4l3k

LGTM

d4l3k · 2025-08-29T17:47:11Z

@pytorchbot merge

pytorchmergebot · 2025-08-29T17:49:53Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2025-08-29T17:55:22Z

Merge failed

Reason: 1 jobs have failed, first few of them are: trunk / macos-py3-arm64 / build

Details for Dev Infra team

Raised by workflow job

d4l3k · 2025-08-29T18:20:07Z

@pytorchbot merge -r viable/strict

pytorchmergebot · 2025-08-29T18:21:53Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2025-08-29T18:21:56Z

Tried to rebase and push PR #156633, but it was already up to date. Try rebasing against main by issuing:
@pytorchbot rebase -b main

pytorchmergebot · 2025-08-29T18:21:56Z

The merge job was canceled or timed out. This most often happen if two merge requests were issued for the same PR, or if merge job was waiting for more than 6 hours for tests to finish. In later case, please do not hesitate to reissue the merge command
For more information see pytorch-bot wiki.

d4l3k · 2025-08-29T18:26:00Z

@pytorchbot rebase -b viable/strict

pytorchmergebot · 2025-08-29T18:27:28Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2025-08-29T18:27:29Z

Tried to rebase and push PR #156633, but it was already up to date. Try rebasing against main by issuing:
@pytorchbot rebase -b main

pytorchmergebot · 2025-08-29T20:18:38Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2025-08-29T20:24:41Z

Merge failed

Reason: 1 jobs have failed, first few of them are: trunk / macos-py3-arm64 / build

Details for Dev Infra team

Raised by workflow job

d4l3k · 2025-09-02T18:08:27Z

@pytorchbot merge

pytorchmergebot · 2025-09-02T18:10:19Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2025-09-02T19:09:41Z

Merge failed

Reason: 1 jobs have failed, first few of them are: trunk / linux-jammy-cuda12.8-py3.10-gcc11 / test (distributed, 2, 3, lf.linux.g4dn.12xlarge.nvidia.gpu)

Details for Dev Infra team

Raised by workflow job

- Enable communication of tensors with Complex datatype in ProcessGroupGloo, similar to how ProcessGroupNCCL handles it. - Move a function, which checks if Complex datatype is supported by a reduce operation, from ProcessGroupNCCL.cpp into a new file to be shared with ProcessGroupGloo. Signed-off-by: Shunzhi Wen <shunzhi.wen@arm.com>

Signed-off-by: Shunzhi Wen <shunzhi.wen@arm.com>

d4l3k · 2025-09-05T16:42:31Z

@pytorchbot merge

pytorchmergebot · 2025-09-05T16:44:30Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

…orch#156633) - Enable communication of tensors with Complex datatype in ProcessGroupGloo, similar to how ProcessGroupNCCL handles it. - Move a function, which checks if Complex datatype is supported by a reduce operation, from ProcessGroupNCCL.cpp into a new file to be shared with ProcessGroupGloo. Fixes pytorch#156632 Pull Request resolved: pytorch#156633 Approved by: https://github.com/d4l3k

pytorch-bot bot added oncall: distributed Add this issue/PR to distributed oncall triage queue release notes: distributed (c10d) release notes category labels Jun 23, 2025

pytorchbot added the open source label Jun 23, 2025

mikaylagawarecki requested a review from d4l3k June 26, 2025 15:55

mikaylagawarecki added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Jun 26, 2025

shunzhiwen force-pushed the gloo_complex_type branch 2 times, most recently from ecf5e06 to 994d85d Compare June 30, 2025 16:19

d4l3k requested changes Aug 11, 2025

View reviewed changes

shunzhiwen force-pushed the gloo_complex_type branch from 994d85d to daf53ed Compare August 18, 2025 19:27

d4l3k approved these changes Aug 19, 2025

View reviewed changes

shunzhiwen force-pushed the gloo_complex_type branch from daf53ed to 3525656 Compare August 28, 2025 16:37

d4l3k approved these changes Aug 28, 2025

View reviewed changes

shunzhiwen force-pushed the gloo_complex_type branch from 3525656 to 7b9f9fc Compare August 29, 2025 17:38

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Aug 29, 2025

pytorchmergebot added the merging label Aug 29, 2025

pytorchmergebot removed the merging label Aug 29, 2025

pytorchmergebot added the merging label Aug 29, 2025

pytorchmergebot removed the merging label Aug 29, 2025

shunzhiwen force-pushed the gloo_complex_type branch from d196937 to c277f9b Compare August 29, 2025 21:08

pytorch-bot bot removed the ciflow/trunk Trigger trunk jobs on your pull request label Aug 29, 2025

shunzhiwen force-pushed the gloo_complex_type branch from c277f9b to bcbb8de Compare September 2, 2025 17:56

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Sep 2, 2025

pytorchmergebot added the merging label Sep 2, 2025

pytorchmergebot removed the merging label Sep 2, 2025

shunzhiwen added 2 commits September 5, 2025 14:56

Add unit tests for complex datatype support in ProcessGroupGloo

5b2c992

Signed-off-by: Shunzhi Wen <shunzhi.wen@arm.com>

shunzhiwen force-pushed the gloo_complex_type branch from bcbb8de to 5b2c992 Compare September 5, 2025 16:25

pytorch-bot bot removed the ciflow/trunk Trigger trunk jobs on your pull request label Sep 5, 2025

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Sep 5, 2025

pytorchmergebot added the merging label Sep 5, 2025

pytorchmergebot added the Merged label Sep 5, 2025

pytorchmergebot closed this in c10195e Sep 5, 2025

pytorchmergebot removed the merging label Sep 5, 2025

[C10d][Gloo] Enable complex datatype support in ProcessGroupGloo #156633

[C10d][Gloo] Enable complex datatype support in ProcessGroupGloo #156633

Uh oh!

Conversation

shunzhiwen commented Jun 23, 2025 • edited by pytorch-bot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Jun 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/156633

✅ No Failures

Uh oh!

linux-foundation-easycla bot commented Jun 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shunzhiwen commented Jun 30, 2025

Uh oh!

d4l3k left a comment

Choose a reason for hiding this comment

Uh oh!

shunzhiwen commented Aug 18, 2025

Uh oh!

d4l3k left a comment

Choose a reason for hiding this comment

Uh oh!

d4l3k Aug 19, 2025

Choose a reason for hiding this comment

Uh oh!

shunzhiwen Aug 28, 2025

Choose a reason for hiding this comment

Uh oh!

d4l3k Aug 19, 2025

Choose a reason for hiding this comment

Uh oh!

shunzhiwen Aug 28, 2025

Choose a reason for hiding this comment

Uh oh!

d4l3k left a comment

Choose a reason for hiding this comment

Uh oh!

d4l3k commented Aug 29, 2025

Uh oh!

pytorchmergebot commented Aug 29, 2025

Merge started

Uh oh!

pytorchmergebot commented Aug 29, 2025

Merge failed

Uh oh!

d4l3k commented Aug 29, 2025

Uh oh!

pytorchmergebot commented Aug 29, 2025

Uh oh!

pytorchmergebot commented Aug 29, 2025

Uh oh!

pytorchmergebot commented Aug 29, 2025

Uh oh!

d4l3k commented Aug 29, 2025

Uh oh!

pytorchmergebot commented Aug 29, 2025

Uh oh!

pytorchmergebot commented Aug 29, 2025

Uh oh!

pytorchmergebot commented Aug 29, 2025

Merge started

Uh oh!

pytorchmergebot commented Aug 29, 2025

Merge failed

Uh oh!

d4l3k commented Sep 2, 2025

Uh oh!

pytorchmergebot commented Sep 2, 2025

Merge started

Uh oh!

pytorchmergebot commented Sep 2, 2025

Merge failed

Uh oh!

d4l3k commented Sep 5, 2025

Uh oh!

pytorchmergebot commented Sep 5, 2025

shunzhiwen commented Jun 23, 2025 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Jun 23, 2025 •

edited

Loading

linux-foundation-easycla bot commented Jun 23, 2025 •

edited

Loading