KEMBAR78
[SymmMem] Add runtime detection of NVSHMEM by kwen2501 · Pull Request #156291 · pytorch/pytorch · GitHub
Skip to content

[SymmMem] Add runtime detection of NVSHMEM #156291

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 7 commits into from
Closed

[SymmMem] Add runtime detection of NVSHMEM #156291

wants to merge 7 commits into from

Conversation

@kwen2501
Copy link
Contributor

@kwen2501 kwen2501 commented Jun 18, 2025 •
edited
Loading

Stack from ghstack (oldest at bottom):

so that we can pick the default backend for SymmetricMemory without
fully relying on env var TORCH_SYMMMEM=CUDA | NVSHMEM

On Python side, the following API is added:
torch.distributed._symmetric_memory.is_nvshmem_available()

cc @H-Huang @awgu @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k

@kwen2501
Update
795a304 
[ghstack-poisoned]
@pytorch-bot
Copy link

pytorch-bot bot commented Jun 18, 2025 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/156291

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (1 Unrelated Failure)

As of commit 6ba7684 with merge base 4d9d884 (image):

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@pytorch-bot pytorch-bot bot added oncall: distributed Add this issue/PR to distributed oncall triage queue release notes: distributed (c10d) release notes category labels Jun 18, 2025
@kwen2501 kwen2501 requested review from fduwjj, fegin and ngimel June 18, 2025 15:29
Skylion007
Skylion007 requested changes Jun 18, 2025
View reviewed changes
torch/csrc/distributed/c10d/init.cpp
py::arg("module"));
#endif

module.def(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: this is a C++ keyword. Not good for future versions.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah I see

In C++, module is a context-sensitive keyword introduced with C++20 for the modules feature, used in declarations like export module MyModule;.

I guess this file needs a big refactor then.

Skylion007
Skylion007 reviewed Jun 18, 2025
View reviewed changes
@kwen2501
Update
e7ae83c 
[ghstack-poisoned]
@kwen2501
Update
9a135a1 
[ghstack-poisoned]
@kwen2501
Update
b085f2e 
[ghstack-poisoned]
kwen2501 added a commit that referenced this pull request Jun 18, 2025
@kwen2501
[SymmMem] Add runtime detection of NVSHMEM
b3fc990 
so that we can pick the default backend for SymmetricMemory without
relying on env var.

ghstack-source-id: 4588d63
Pull-Request-resolved: #156291
@kwen2501
Update
8b26449 
[ghstack-poisoned]
kwen2501 added a commit that referenced this pull request Jun 18, 2025
@kwen2501
[SymmMem] Add runtime detection of NVSHMEM
548567f 
so that we can pick the default backend for SymmetricMemory without
relying on env var.

ghstack-source-id: b4cc209
Pull-Request-resolved: #156291
@kwen2501
Update
c81c67b 
[ghstack-poisoned]
kwen2501 added a commit that referenced this pull request Jun 19, 2025
@kwen2501
[SymmMem] Add runtime detection of NVSHMEM
548b7b8 
so that we can pick the default backend for SymmetricMemory without
relying on env var.

ghstack-source-id: de17da3
Pull-Request-resolved: #156291
@kwen2501
Update
6ba7684 
[ghstack-poisoned]
kwen2501 added a commit that referenced this pull request Jun 19, 2025
@kwen2501
[SymmMem] Add runtime detection of NVSHMEM
b9932a8 
so that we can pick the default backend for SymmetricMemory without
relying on env var.

ghstack-source-id: 2941a79
Pull-Request-resolved: #156291
@kwen2501 kwen2501 added the ciflow/trunk Trigger trunk jobs on your pull request label Jun 19, 2025
@kwen2501
Copy link
Contributor Author

@pytorchbot merge

@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@kwen2501 kwen2501 mentioned this pull request Jun 25, 2025
Closed
@github-actions github-actions bot deleted the gh/kwen2501/175/head branch July 20, 2025 02:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Skylion007 Skylion007 Skylion007 approved these changes

@fegin fegin fegin left review comments

@ngimel ngimel Awaiting requested review from ngimel

@fduwjj fduwjj Awaiting requested review from fduwjj

Assignees

No one assigned

Labels

ciflow/trunk Trigger trunk jobs on your pull request Merged oncall: distributed Add this issue/PR to distributed oncall triage queue release notes: distributed (c10d) release notes category

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@kwen2501 @pytorchmergebot @Skylion007 @fegin