🐛 Describe the bug

I am testing gradients with torch.autograd.gradcheck.gradcheck. If I am setting the default device to my GPU with torch.set_default_tensor_type("torch.cuda.FloatTensor"), I get a device mismatch. The error only occurs in fast mode, not in slow mode. The problem comes from:

It seems like the Generator is not given a device and uses the CPU. Similar issues also identified the Generator as the problem and solved it by simply passing "cuda" (e.g. here).

In gradcheck.py, however, one cannot access the generator, and it is also explicitly named "g_cpu". Is gradcheck in fast mode not meant to be run on GPU? Did I miss anything?

Versions

I am running version 1.12.1, but since the offending piece of code is still the same in the current main branch, it should still be an issue.

cc @ezyang @albanD @zou3519 @gqchen @pearu @nikitaved @soulitzer @lezcano @Varal7