[c10d] Use TORCH_CHECK for monitored barrier error (#59667)

rohan-varma · facebook-github-bot · commit fc0582ee9516 · 2021-06-09T22:31:33.000-07:00
Summary: Pull Request resolved: #59667 Use torch_check over throw std::runtime_error in monitored barrier so that it works with torch_cpp_show_stacktraces to reveal the entire callstack where the monitored barrier failed, which can help determine where the particular rank encountered an issue. ghstack-source-id: 130993689 Test Plan: CI Reviewed By: cbalioglu Differential Revision: D28974510 fbshipit-source-id: 6a6958995c1066cddcd647ca88c74473079b69fc
diff --git a/torch/lib/c10d/ProcessGroupGloo.cpp b/torch/lib/c10d/ProcessGroupGloo.cpp
@@ -136,12 +136,12 @@ std::chrono::milliseconds getRemainingTime(
   return remainingMillis;
 }
 
-// Emit a LOG(ERROR) and throws runtime_error with the given messages.
+// Emit a LOG(ERROR) and throws using TORCH_CHECK with the given messages.
 void logAndThrow(
     const std::string& logMessage,
     const std::string& errorMessage) {
   LOG(ERROR) << logMessage;
-  throw std::runtime_error(errorMessage);
+  TORCH_CHECK(false, errorMessage);
 }
 
 // For monitoredBarrier, checks remaining time left to finish processing ranks
