🚀 The feature, motivation and pitch

As of 1.12, only limited shared param support exists for FSDP, i.e., they must be part of the same FSDP unit, user cannot shared parameters if their respective modules end up being wrapped as part of different FSDP units. Creating this issue to track full support.

Alternatives

No response

Additional context

No response

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @kwen2501