Multiprocessing support for NT #110292

jbschlosser · 2023-09-29T17:36:08Z

Stack from ghstack (oldest at bottom):

Fixes #110161

Allows NTs to be used in DataLoaders with num_workers > 1.

[ghstack-poisoned]

pytorch-bot · 2023-09-29T17:36:12Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/110292

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (2 Unrelated Failures)

As of commit a514685 with merge base 46a5558 ():

UNSTABLE - The following jobs failed but were likely due to flakiness present on trunk and has been marked as unstable:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

ghstack-source-id: 710581d Pull Request resolved: #110292

jbschlosser · 2023-09-29T18:57:46Z

@pytorchbot merge

pytorchmergebot · 2023-09-29T18:59:48Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2023-09-29T19:32:03Z

Merge failed

Reason: 1 jobs have failed, first few of them are: trunk / macos-12-py3-arm64 / test (default, 1, 3, macos-m1-12)

Details for Dev Infra team

Raised by workflow job

cpuhrsch · 2023-09-30T07:28:01Z

Failures look real unfortunately @jbschlosser

Fixes #110161 Allows NTs to be used in DataLoaders with `num_workers > 1`. [ghstack-poisoned]

jbschlosser · 2023-10-02T16:47:11Z

@pytorchbot merge

pytorchmergebot · 2023-10-02T16:48:57Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

torch/multiprocessing/reductions.py

pytorch-bot · 2023-10-02T18:23:46Z

❌ 🤖 pytorchbot command failed:

@pytorchbot revert: error: the following arguments are required: -m/--message, -c/--classification

usage: @pytorchbot revert -m MESSAGE -c
                          {nosignal,ignoredsignal,landrace,weird,ghfirst}

Try @pytorchbot --help for more info.

pytorch-bot · 2023-10-02T18:24:05Z

❌ 🤖 pytorchbot command failed:

@pytorchbot revert: error: the following arguments are required: -c/--classification

usage: @pytorchbot revert -m MESSAGE -c
                          {nosignal,ignoredsignal,landrace,weird,ghfirst}

Try @pytorchbot --help for more info.

jbschlosser · 2023-10-02T18:24:42Z

@pytorchbot revert -m "Address review comments" -c "weird"

pytorchmergebot · 2023-10-02T18:27:07Z

@pytorchbot successfully started a revert job. Check the current status here.
Questions? Feedback? Please reach out to the PyTorch DevX Team

Fixes #110161 Allows NTs to be used in DataLoaders with `num_workers > 1`. [ghstack-poisoned]

jbschlosser · 2023-10-04T21:44:18Z

@pytorchbot merge

pytorchmergebot · 2023-10-04T21:47:50Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2023-10-04T22:08:51Z

Merge failed

Reason: 1 jobs have failed, first few of them are: trunk / macos-12-py3-arm64 / build

Details for Dev Infra team

Raised by workflow job

jbschlosser · 2023-10-05T15:02:43Z

@pytorchbot merge -f "ignore spurious failure"

pytorchmergebot · 2023-10-05T15:04:35Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

Pull Request resolved: #110404 Approved by: https://github.com/cpuhrsch, https://github.com/albanD ghstack dependencies: #110292

kit1980 · 2023-10-06T01:04:52Z

@pytorchbot revert -m "Causes CUDA memory leaks" -c nosignal

RuntimeError: CUDA driver API confirmed a leak in main.TestDataLoaderDeviceTypeCUDA.test_nested_tensor_multiprocessing_context_forkserver_cuda! Caching allocator allocated memory was 5120 and is now reported as 10240 on device 0. CUDA driver allocated memory was 340459520 and is now 342556672.

https://github.com/pytorch/pytorch/actions/runs/6425541384/job/17449001020

pytorchmergebot · 2023-10-06T01:07:34Z

@pytorchbot successfully started a revert job. Check the current status here.
Questions? Feedback? Please reach out to the PyTorch DevX Team

pytorchmergebot · 2023-10-06T01:07:44Z

@jbschlosser your PR has been successfully reverted.

This reverts commit f17fe89. Reverted #110292 on behalf of https://github.com/kit1980 due to Causes CUDA memory leaks ([comment](#110292 (comment)))

Fixes #110161 Allows NTs to be used in DataLoaders with `num_workers > 1`. [ghstack-poisoned]

jbschlosser · 2023-10-10T19:00:11Z

@pytorchbot merge

pytorchmergebot · 2023-10-10T19:02:32Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

Pull Request resolved: #110404 Approved by: https://github.com/cpuhrsch, https://github.com/albanD ghstack dependencies: #110292

Multiprocessing support for NT

714e8a8

[ghstack-poisoned]

pytorch-bot bot added the release notes: dataloader release notes category label Sep 29, 2023

jbschlosser added a commit that referenced this pull request Sep 29, 2023

Multiprocessing support for NT

0a0e7db

ghstack-source-id: 710581d Pull Request resolved: #110292

jbschlosser added topic: improvements topic category release notes: nested tensor Changes that have a direct impact on nested tensors and removed release notes: dataloader release notes category labels Sep 29, 2023

jbschlosser requested a review from cpuhrsch September 29, 2023 17:38

cpuhrsch approved these changes Sep 29, 2023

View reviewed changes

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Sep 29, 2023

pytorchmergebot added the merging label Sep 29, 2023

pytorchmergebot removed the merging label Sep 29, 2023

Update on "Multiprocessing support for NT"

f00bd65

Fixes #110161 Allows NTs to be used in DataLoaders with `num_workers > 1`. [ghstack-poisoned]

pytorchmergebot added the merging label Oct 2, 2023

albanD reviewed Oct 2, 2023

View reviewed changes

torch/multiprocessing/reductions.py Outdated Show resolved Hide resolved

jbschlosser removed the merging label Oct 2, 2023

pytorchmergebot added the Merged label Oct 2, 2023

pytorchmergebot closed this in 881e730 Oct 2, 2023

Update on "Multiprocessing support for NT"

3939e2f

Fixes #110161 Allows NTs to be used in DataLoaders with `num_workers > 1`. [ghstack-poisoned]

pytorchmergebot added the merging label Oct 4, 2023

pytorchmergebot removed the merging label Oct 4, 2023

pytorchmergebot added the merging label Oct 5, 2023

pytorchmergebot removed the merging label Oct 5, 2023

pytorchmergebot closed this in f17fe89 Oct 5, 2023

pytorchmergebot pushed a commit that referenced this pull request Oct 5, 2023

pin_memory support for NT (#110404)

3597325

Pull Request resolved: #110404 Approved by: https://github.com/cpuhrsch, https://github.com/albanD ghstack dependencies: #110292

This was referenced Oct 5, 2023

Make unbind() overrideable for NT subclass #110646

Closed

Implement split_with_sizes backward for NT #110647

Closed

pytorchmergebot added a commit that referenced this pull request Oct 6, 2023

Revert "Multiprocessing support for NT (#110292)"

dac895c

This reverts commit f17fe89. Reverted #110292 on behalf of https://github.com/kit1980 due to Causes CUDA memory leaks ([comment](#110292 (comment)))

jbschlosser reopened this Oct 6, 2023

Update on "Multiprocessing support for NT"

a514685

Fixes #110161 Allows NTs to be used in DataLoaders with `num_workers > 1`. [ghstack-poisoned]

jbschlosser requested a review from soulitzer as a code owner October 10, 2023 18:23

pytorchmergebot added the merging label Oct 10, 2023

pytorchmergebot removed the merging label Oct 10, 2023

pytorchmergebot closed this in 43ea782 Oct 10, 2023

pytorchmergebot pushed a commit that referenced this pull request Oct 10, 2023

pin_memory support for NT (#110404)

ac01304

Pull Request resolved: #110404 Approved by: https://github.com/cpuhrsch, https://github.com/albanD ghstack dependencies: #110292

facebook-github-bot deleted the gh/jbschlosser/91/head branch October 14, 2023 14:24

Multiprocessing support for NT #110292

Multiprocessing support for NT #110292

Uh oh!

Conversation

jbschlosser commented Sep 29, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Sep 29, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/110292

✅ You can merge normally! (2 Unrelated Failures)

Uh oh!

jbschlosser commented Sep 29, 2023

Uh oh!

pytorchmergebot commented Sep 29, 2023

Merge started

Uh oh!

pytorchmergebot commented Sep 29, 2023

Merge failed

Uh oh!

cpuhrsch commented Sep 30, 2023

Uh oh!

jbschlosser commented Oct 2, 2023

Uh oh!

pytorchmergebot commented Oct 2, 2023

Merge started

Uh oh!

Uh oh!

pytorch-bot bot commented Oct 2, 2023

Uh oh!

pytorch-bot bot commented Oct 2, 2023

Uh oh!

jbschlosser commented Oct 2, 2023

Uh oh!

pytorchmergebot commented Oct 2, 2023

Uh oh!

jbschlosser commented Oct 4, 2023

Uh oh!

pytorchmergebot commented Oct 4, 2023

Merge started

Uh oh!

pytorchmergebot commented Oct 4, 2023

Merge failed

Uh oh!

jbschlosser commented Oct 5, 2023

Uh oh!

pytorchmergebot commented Oct 5, 2023

Merge started

Uh oh!

kit1980 commented Oct 6, 2023

Uh oh!

pytorchmergebot commented Oct 6, 2023

Uh oh!

pytorchmergebot commented Oct 6, 2023

Uh oh!

jbschlosser commented Oct 10, 2023

Uh oh!

pytorchmergebot commented Oct 10, 2023

Merge started

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jbschlosser commented Sep 29, 2023 •

edited

Loading

pytorch-bot bot commented Sep 29, 2023 •

edited

Loading