[pipelining] allow multiple backward grads #140981
Conversation
[ghstack-poisoned]
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/140981
Note: Links to docs will display an error until the docs builds have been completed. ✅ You can merge normally! (1 Unrelated Failure)As of commit 9b5b50d with merge base b379a28 ( FLAKY - The following job failed but was likely due to flakiness present on trunk:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
pytorch-bot
bot
added
the
oncall: distributed
H-Huang
requested review from
kwen2501 and
wconstab
and removed request for
kwen2501
| _Action(2, F, 1), | ||
| None, | ||
| None, | ||
| _Action(2, B, 0), |
There was a problem hiding this comment.
i'm probably just missing something but isn't this schedule ordered in the usual order, i.e. not reordered B?
the usual order would execute B's in increasing microbatch_id sequence right?
There was a problem hiding this comment.
Yeah good point, I realized the test is not good because the error would only surface if we use send for mb A after we did a backward for mb B, which the test is not catching. So i updated it to reorder the send operations in the new schedule
fixes #139404. The input grads get saved in a new `self.bwd_cache` container and get popped off after they are used in `backward_one_chunk` `python test/distributed/pipelining/test_schedule_multiproc.py -k test_pipeline_schedule_runtime_custom_sched` cc awgu kwen2501 wanchaol fegin fduwjj wz337 wconstab d4l3k c-p-i-o [ghstack-poisoned]
|
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Merge failedReason: 1 mandatory check(s) failed. The first few are: Dig deeper by viewing the failures on hud |
fixes #139404. The input grads get saved in a new `self.bwd_cache` container and get popped off after they are used in `backward_one_chunk` `python test/distributed/pipelining/test_schedule_multiproc.py -k test_pipeline_schedule_runtime_custom_sched` cc awgu kwen2501 wanchaol fegin fduwjj wz337 wconstab d4l3k c-p-i-o [ghstack-poisoned]
|
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
fixes pytorch#139404. The input grads get saved in a new `self.bwd_cache` container and get popped off after they are used in `backward_one_chunk` `python test/distributed/pipelining/test_schedule_multiproc.py -k test_pipeline_schedule_runtime_custom_sched` Pull Request resolved: pytorch#140981 Approved by: https://github.com/wconstab
fixes pytorch#139404. The input grads get saved in a new `self.bwd_cache` container and get popped off after they are used in `backward_one_chunk` `python test/distributed/pipelining/test_schedule_multiproc.py -k test_pipeline_schedule_runtime_custom_sched` Pull Request resolved: pytorch#140981 Approved by: https://github.com/wconstab
3 participants
Stack from ghstack (oldest at bottom):
fixes #139404. The input grads get saved in a new
self.bwd_cachecontainer and get popped off after they are used inbackward_one_chunkpython test/distributed/pipelining/test_schedule_multiproc.py -k test_pipeline_schedule_runtime_custom_schedcc @awgu @kwen2501 @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k @c-p-i-o