[FSDP2] Use reduceOpSum for world size 1 #157529
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/157529
Note: Links to docs will display an error until the docs builds have been completed. ✅ You can merge normally! (1 Unrelated Failure)As of commit 4f2fa3e with merge base f56bfb3 ( FLAKY - The following job failed but was likely due to flakiness present on trunk:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
pytorch-bot
bot
added
ciflow/inductor
oncall: distributed
mori360
changed the title
|
|
||
| from torch.distributed.distributed_c10d import ReduceOp | ||
|
|
||
| model = ModelOfModel() |
There was a problem hiding this comment.
why do we invent ModelOfModel for world_size 1 unit test?
| dist.reduce_scatter_tensor( | ||
| output=reduce_output_sum, input=reduce_scatter_input, op=ReduceOp.SUM | ||
| ) | ||
| self.assertNotEqual(reduce_output_avg, reduce_scatter_input) |
There was a problem hiding this comment.
why we need to assert NotEqual for avg? it's a nccl bug
| self.assertEqual(reduce_output_sum, reduce_scatter_input) | ||
|
|
||
| model = MLP(4) | ||
| fully_shard(model) |
There was a problem hiding this comment.
I think your previous unit test is right. just need to replca modelofmodel with this simple MLP
|
@pytorchmergebot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Merge failedReason: 1 jobs have failed, first few of them are: trunk / linux-jammy-rocm-py3.10 / test (default, 2, 2, linux.rocm.gpu.2) Details for Dev Infra teamRaised by workflow job |
|
@pytorchmergebot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
5 participants
cc @H-Huang @awgu @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k