[FSDP] Use post_reduce_stream.record_event() on hsdp+cpuoffload #160481
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/160481
Note: Links to docs will display an error until the docs builds have been completed. ✅ You can merge normally! (1 Unrelated Failure)As of commit 5002451 with merge base 211c988 ( BROKEN TRUNK - The following job failed but were present on the merge base:👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
pytorch-bot
bot
added
ciflow/inductor
oncall: distributed
mori360
changed the title
|
@pytorchbot rebase |
|
@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here |
| ) | ||
| model = Transformer(model_args) | ||
| ref_model = copy.deepcopy(model) | ||
| if device_type == device_type: |
There was a problem hiding this comment.
device_type == device_type here is always true
There was a problem hiding this comment.
one of the device_type is referring to https://github.com/pytorch/pytorch/blob/82c7a1eb4b743408e907bdd09ea5645af964ae85/test/distributed/_composable/fsdp/test_fully_shard_training.py#L60C1-L60C12
rename device_type in this unit test?
|
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Merge failedReason: 1 jobs have failed, first few of them are: inductor / cuda12.8-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) Details for Dev Infra teamRaised by workflow job |
|
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
…rch#160481) Fixes pytorch#160291 `post_reduce_stream` is `all_reduce_stream` during HSDP, but CPU-GPU sync is hard coded to `reduce_scatter_stream` The hard-code could fail unit test on HSDP+CPU offload, add unit test here. Pull Request resolved: pytorch#160481 Approved by: https://github.com/weifengpy
…rch#160481) Fixes pytorch#160291 `post_reduce_stream` is `all_reduce_stream` during HSDP, but CPU-GPU sync is hard coded to `reduce_scatter_stream` The hard-code could fail unit test on HSDP+CPU offload, add unit test here. Pull Request resolved: pytorch#160481 Approved by: https://github.com/weifengpy
3 participants
Fixes #160291
post_reduce_streamisall_reduce_streamduring HSDP, but CPU-GPU sync is hard coded toreduce_scatter_streamThe hard-code could fail unit test on HSDP+CPU offload, add unit test here.
cc @H-Huang @awgu @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k @pragupta