Update on "Change ProcessGroupNCCL default timeout to 10 min"

wconstab · wconstab · commit 9aa9a395a259 · 2023-10-10T11:19:47.000-07:00
Avoid changing default for other backends as CPU backend (GLOO) may need
longer timeouts.

Motivated by trying to save cluster time when encountering collective
hangs.  Generally collectives should time out within seconds and 30
minutes (or 10 minutes) should provide ample headroom for edge cases.

[ghstack-poisoned]
diff --git a/torch/csrc/distributed/c10d/ProcessGroupNCCL.hpp b/torch/csrc/distributed/c10d/ProcessGroupNCCL.hpp
@@ -179,6 +179,7 @@ class TORCH_API ProcessGroupNCCL : public Backend {
     // In case of timeout, set exception on the WorkNCCL object.
     bool checkTimeout(
         c10::optional<std::chrono::milliseconds> timeout = c10::nullopt);
+
     std::vector<at::Tensor> result() override;
 
    protected:
