fix(bigquery): Make exponential backoff retry second based (#2212)

mina-asham · gcf-owl-bot[bot] · web-flow · commit eff4a09aece9 · 2023-11-01T15:51:45.000-07:00
* fix(bigquery): Make exponential backoff retry second based - Currently these retries are doing retries in milliseconds, which is way too aggressive, the current retry steps are (1ms, 2ms, 4ms, 8ms, 16ms, 32ms, 64ms, 128ms, 256ms, 512ms, 1024ms, ...), it's highly likely the first 7 retries will fail with rate limiting specially in a big workload (specifically this is causing an issue in the Spark bigquery connector), I suspect the exponential retries here were meant to be in seconds (though even if it wasn't meant to, this might be a slightly better approach to reduce the load on the servers - The new steps would be: 1s, 2s, 4s, 8s, 16s, 32s, 60s (repeated till we exhaust the 5 minutes max retries) * Update retry base to 50ms * Update tests * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Avoid long overflow - Make the comparison happen on the double side then cast to long at the end to avoid overflow happening from the calculation itself - This was actually a bug from before, just wasn't happening because there was no 50 multiplier to get it into that state! --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
diff --git a/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/ConnectionWorker.java b/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/ConnectionWorker.java
@@ -590,7 +590,7 @@ private void maybeWaitForInflightQuota() {
 
   @VisibleForTesting
   static long calculateSleepTimeMilli(long retryCount) {
-    return Math.min((long) Math.pow(2, retryCount), 60000);
+    return (long) Math.min(Math.pow(2, retryCount) * 50, 60000);
   }
 
   @VisibleForTesting
diff --git a/google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1/ConnectionWorkerTest.java b/google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1/ConnectionWorkerTest.java
@@ -518,8 +518,8 @@ public void testStreamNameMismatch() throws Exception {
 
   @Test
   public void testExponentialBackoff() throws Exception {
-    assertThat(ConnectionWorker.calculateSleepTimeMilli(0)).isEqualTo(1);
-    assertThat(ConnectionWorker.calculateSleepTimeMilli(5)).isEqualTo(32);
+    assertThat(ConnectionWorker.calculateSleepTimeMilli(0)).isEqualTo(50);
+    assertThat(ConnectionWorker.calculateSleepTimeMilli(5)).isEqualTo(1600);
     assertThat(ConnectionWorker.calculateSleepTimeMilli(100)).isEqualTo(60000);
   }
 
