Tracking bug: com.google.cloud.storage.StorageException: Connection closed prematurely:.. #165
Description
Using a ReadChannel, if a premature connection closure occurs, the client isn't able to recover when it should be able to. The client at that moment has a specific range context which makes it retryable.
Additionally, the implementation is too reliant on etag1 not changing when generation number can do the same thing. This also needs to be addressed.
Example failure:
CONFIG: -------------- REQUEST --------------
GET http://localhost:8080/download/storage/v1/b/test-bucket/o/test-create-blob?alt=media
Accept-Encoding: gzip
Authorization: <Not Logged>
Range: bytes=0-10485759
User-Agent: gcloud-java/ Google-API-Java-Client/1.30.9 Google-HTTP-Java-Client/1.34.2 (gzip)
x-goog-api-client: gl-java/1.8.0 gdcl/1.30.9 linux/5.2.17
x-goog-testbench-instructions: return-broken-stream
Mar 06, 2020 8:47:01 AM com.google.api.client.http.HttpRequest execute
CONFIG: curl -v --compressed -H 'Accept-Encoding: gzip' -H 'Authorization: <Not Logged>' -H 'Range: bytes=0-10485759' -H 'User-Agent: gcloud-java/ Google-API-Java-Client/1.30.9 Google-HTTP-Java-Client/1.34.2 (gzip)' -H 'x-goog-api-client: gl-java/1.8.0 gdcl/1.30.9 linux/5.2.17' -H 'x-goog-testbench-instructions: return-broken-stream' -- 'http://localhost:8080/download/storage/v1/b/test-bucket/o/test-create-blob?alt=media'
Mar 06, 2020 8:47:01 AM com.google.api.client.http.HttpResponse <init>
CONFIG: -------------- RESPONSE --------------
HTTP/1.0 200 OK
Server: Werkzeug/1.0.0 Python/3.7.0
Content-Range: bytes 0-10485758/10485760
x-goog-generation: 1
Content-Length: 10485760
Date: Fri, 06 Mar 2020 16:47:01 GMT
Content-Type: text/html; charset=utf-8
x-goog-hash: md5=8clkXbwU793H2KMiaF8m6w==,crc32c=CPJwmw==
Mar 06, 2020 8:47:03 AM com.google.api.client.util.LoggingByteArrayOutputStream close
CONFIG: Total: 1,114,112 bytes (logging first 16,384 bytes)
Mar 06, 2020 8:47:03 AM com.google.api.client.util.LoggingByteArrayOutputStream close
CONFIG:
com.google.cloud.storage.StorageException: Connection closed prematurely: bytesRead = 1114112, Content-Length = 10485760
at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:230)
at com.google.cloud.storage.spi.v1.HttpStorageRpc.read(HttpStorageRpc.java:711)
at com.google.cloud.storage.BlobReadChannel$1.call(BlobReadChannel.java:129)
at com.google.cloud.storage.BlobReadChannel$1.call(BlobReadChannel.java:125)
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105)
at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
at com.google.cloud.storage.BlobReadChannel.read(BlobReadChannel.java:124)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at com.google.cloud.storage.it.ITStorageTest.testFailure(ITStorageTest.java:359)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
Caused by: java.io.IOException: Connection closed prematurely: bytesRead = 1114112, Content-Length = 10485760
at com.google.api.client.http.javanet.NetHttpResponse$SizeValidatingInputStream.throwIfFalseEOF(NetHttpResponse.java:204)
at com.google.api.client.http.javanet.NetHttpResponse$SizeValidatingInputStream.read(NetHttpResponse.java:166)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at com.google.api.client.util.LoggingInputStream.read(LoggingInputStream.java:57)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at com.google.api.client.util.ByteStreams.copy(ByteStreams.java:49)
at com.google.api.client.util.IOUtils.copy(IOUtils.java:87)
at com.google.api.client.util.IOUtils.copy(IOUtils.java:59)
at com.google.api.client.http.HttpResponse.download(HttpResponse.java:410)
at com.google.cloud.storage.spi.v1.HttpStorageRpc.read(HttpStorageRpc.java:706)
... 40 more
Using C++ testbench2 to reproduce issue but requires tweaks3 to work with the Java library.