🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/138021

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 93d4434 with merge base 7435f57 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

The committers listed above are authorized under a signed CLA.

• ✅ login: samnordmann (93d4434, d62c095)

just for my curiosity, what is the motivation for using UCC? (compared to Gloo, perhaps)

just for my curiosity, what is the motivation for using UCC? (compared to Gloo, perhaps)

UCC fully supports host and device memory, many different transports (IB, Sharp, mlx5dv, cuda, ROCm, shared memory, etc.), and achieves state-of-the-art performance, with low GPU SM usage.
https://ieeexplore.ieee.org/document/10664373

Hi, first, thanks for the contribution.
We are thinking of periodically reviewing torch's hosting of backend implementations.
If you could share some libraries / users that are using ProcessGroupUCC (that are okay to be publicly known), that would help with the review.
Thanks!

Hi, first, thanks for the contribution. We are thinking of periodically reviewing torch's hosting of backend implementations. If you could share some libraries / users that are using ProcessGroupUCC (that are okay to be publicly known), that would help with the review. Thanks!

Hi, here is a (probably non-exhaustive) list of other libraries using ProcessGroupUCC

• NEMO - Enabling UCC backend for PP communication by youngeunkwon0405 · Pull Request #10531 · NVIDIA/NeMo
• Megatron - Enabling UCC backend for PP communication by youngeunkwon0405 · Pull Request #1157 · NVIDIA/Megatron-LM
• nvFuser https://github.com/NVIDIA/Fuser/blob/96d64b61612f3248062c7d17c68582f94b92e37c/csrc/multidevice/communicator.cpp#L165
• PARAM - Support benchmarking UCC backend through PyTorch in PARAM-Comms by kingchc · Pull Request #14 · facebookresearch/param
• DLRM by Meta - Added torch_ucc support by artemry-nv · Pull Request #173 · facebookresearch/dlrm (https://ucfconsortium.org/wp-content/uploads/2021/02/Joshua_Ladd_UCC_PG_UCFvF2F_Dec2020_v7_jss.pdf)
• Mixing AMD and NVIDIA GPUs - Question about failing collective allreduce across on non-homogenous ring (NVIDIA and AMD GPU) · Issue #1043 · openucx/ucc

@kwen2501 @wconstab is it ready to be merged? Or do you have any indication on the next step?

@pytorchbot rebase

You don't have permissions to rebase this PR since you are a first time contributor. If you think this is a mistake, please contact PyTorch Dev Infra.

@pytorchbot rebase

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

Successfully rebased add_ucc_reduce_scatter_base onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout add_ucc_reduce_scatter_base && git pull --rebase)

@pytorchbot merge

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team