-
Notifications
You must be signed in to change notification settings - Fork 1.7k
Closed
Labels
Description
Environment information (required)
--- check: autoidentify
INFO: diagnose_tensorboard.py version 724b56cee52e7d8eb89bbeec1f0d5ce3e38c9682
--- check: general
INFO: sys.version_info: sys.version_info(major=3, minor=6, micro=8, releaselevel='final', serial=0)
INFO: os.name: posix
INFO: os.uname(): posix.uname_result(sysname='Darwin', release='19.0.0', version='Darwin Kernel Version 19.0.0: Thu Oct 17 16:17:15 PDT 2019; root:xnu-6153.41.3~29/RELEASE_X86_64', machine='x86_64')
--- check: package_management
INFO: has conda-meta: True
INFO: $VIRTUAL_ENV: None
--- check: installed_packages
INFO: installed: tensorboard==2.2.1
--- check: full_pip_freeze
boto3==1.13.1
botocore==1.16.1
Issue description
tensorboard/compat/tensorflow_stub/io/gfile.py:292 identifies out of range errors with "416" whereas above mentioned s3 client returns "InvalidRange". This leads to reraising the exception when served from S3.
I set up a public bucket for testing (rate limits might cause probs):
AWS_ACCESS_KEY_ID=002ee7e2d071a250000000003 AWS_SECRET_ACCESS_KEY=K002Obc5W+Oc/nE8sFsCDYnOxMx/Xxk S3_ENDPOINT=https://s3.us-west-002.backblazeb2.com tensorboard --logdir s3://pubtests/tensorboards/tb2
Error:
Traceback (most recent call last):
File "~/miniconda3/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "~/miniconda3/lib/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "~/miniconda3/lib/python3.6/site-packages/tensorboard/backend/application.py", line 727, in _reload
multiplexer.Reload()
File "~/miniconda3/lib/python3.6/site-packages/tensorboard/backend/event_processing/plugin_event_multiplexer.py", line 259, in Reload
Worker()
File "~/miniconda3/lib/python3.6/site-packages/tensorboard/backend/event_processing/plugin_event_multiplexer.py", line 237, in Worker
accumulator.Reload()
File "~/miniconda3/lib/python3.6/site-packages/tensorboard/backend/event_processing/plugin_event_accumulator.py", line 183, in Reload
for event in self._generator.Load():
File "~/miniconda3/lib/python3.6/site-packages/tensorboard/backend/event_processing/directory_watcher.py", line 90, in Load
for event in self._LoadInternal():
File "~/miniconda3/lib/python3.6/site-packages/tensorboard/backend/event_processing/directory_watcher.py", line 120, in _LoadInternal
for event in self._loader.Load():
File "~/miniconda3/lib/python3.6/site-packages/tensorboard/backend/event_processing/event_file_loader.py", line 164, in Load
for record in super(EventFileLoader, self).Load():
File "~/miniconda3/lib/python3.6/site-packages/tensorboard/backend/event_processing/event_file_loader.py", line 139, in Load
yield next(self._iterator)
File "~/miniconda3/lib/python3.6/site-packages/tensorboard/backend/event_processing/event_file_loader.py", line 110, in __next__
self._reader.GetNext()
File "~/miniconda3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/pywrap_tensorflow.py", line 210, in GetNext
header_str = self._read(8)
File "~/miniconda3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/pywrap_tensorflow.py", line 274, in _read
new_data = self.file_handle.read(n)
File "~/miniconda3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/io/gfile.py", line 498, in read
self.filename, self.binary_mode, read_size, self.continuation_token
File "~/miniconda3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/io/gfile.py", line 290, in read
stream = s3.Object(bucket, path).get(**args)["Body"].read()
File "~/miniconda3/lib/python3.6/site-packages/boto3/resources/factory.py", line 520, in do_action
response = action(self, *args, **kwargs)
File "~/miniconda3/lib/python3.6/site-packages/boto3/resources/action.py", line 83, in __call__
response = getattr(parent.meta.client, operation_name)(*args, **params)
File "~/miniconda3/lib/python3.6/site-packages/botocore/client.py", line 316, in _api_call
return self._make_api_call(operation_name, kwargs)
File "~/miniconda3/lib/python3.6/site-packages/botocore/client.py", line 626, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidRange) when calling the GetObject operation: byte range bytes=40-16777256 for a file of length 40
Alexfinkelshtein