would love to support more from pytorch fsdp2 side @gshennvm @yuki-666 @terrykong @parthchadha

would love to support more from pytorch fsdp2 side @gshennvm @yuki-666 @terrykong @parthchadha

thanks for the contribution! Just for my understanding -- doesn't pytorch set this to False by default already?

from the pytorch docs:

The root FSDP state has its value specially set to False as a heuristic since its parameters would typically be immediately all-gathered for backward.

cc @terrykong on how we can work together for better fsdp2 integration :)

doesn't pytorch set this to False by default already?

from the pytorch docs:

The root FSDP state has its value specially set to False as a heuristic since its parameters would typically be immediately all-gathered for backward.

I am about to remove the heuristic for future pytorch release: pytorch/pytorch#154704

for root model, if user set fully_shard(reshard_after_forward=True),

• in existing pytorch release, we will override it to False. it's too implicit
• in future pytorch release, we respect user's config on reshard_after_forward=True

I will change the doc as well to recommend reshard_after_forward=False for root model

Hi @weifengpy . Thanks, could you rebase your commits to --signoff to pass our DCO check?

Hi @weifengpy . Thanks, could you rebase your commits to --signoff to pass our DCO check?

just commited with --signoff