set_default_device/torch.device has performance impact for non-factory functions #92701
Description
🐛 Describe the bug
Because it is implemented as a TorchFunctionMode, we interpose ALL functions even if we actually do nothing on those functions. We have ideas for how to optimize this but have delayed doing so until someone actually has a problem. This is a placeholder issue to track people having this problem.
Versions
master
cc @hameerabbasi @rgommers @peterbell10 @soumith @msaroufim @wconstab @ngimel @bdhirsh