KEMBAR78
S3 range error is not caught · Issue #3608 · tensorflow/tensorboard · GitHub
Skip to content

S3 range error is not caught #3608

@ahirner

Description

@ahirner

Environment information (required)

--- check: autoidentify
INFO: diagnose_tensorboard.py version 724b56cee52e7d8eb89bbeec1f0d5ce3e38c9682

--- check: general
INFO: sys.version_info: sys.version_info(major=3, minor=6, micro=8, releaselevel='final', serial=0)
INFO: os.name: posix
INFO: os.uname(): posix.uname_result(sysname='Darwin', release='19.0.0', version='Darwin Kernel Version 19.0.0: Thu Oct 17 16:17:15 PDT 2019; root:xnu-6153.41.3~29/RELEASE_X86_64', machine='x86_64')

--- check: package_management
INFO: has conda-meta: True
INFO: $VIRTUAL_ENV: None

--- check: installed_packages
INFO: installed: tensorboard==2.2.1

--- check: full_pip_freeze
boto3==1.13.1
botocore==1.16.1

Issue description

tensorboard/compat/tensorflow_stub/io/gfile.py:292 identifies out of range errors with "416" whereas above mentioned s3 client returns "InvalidRange". This leads to reraising the exception when served from S3.

I set up a public bucket for testing (rate limits might cause probs):
AWS_ACCESS_KEY_ID=002ee7e2d071a250000000003 AWS_SECRET_ACCESS_KEY=K002Obc5W+Oc/nE8sFsCDYnOxMx/Xxk S3_ENDPOINT=https://s3.us-west-002.backblazeb2.com tensorboard --logdir s3://pubtests/tensorboards/tb2

Error:

Traceback (most recent call last):
  File "~/miniconda3/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "~/miniconda3/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "~/miniconda3/lib/python3.6/site-packages/tensorboard/backend/application.py", line 727, in _reload
    multiplexer.Reload()
  File "~/miniconda3/lib/python3.6/site-packages/tensorboard/backend/event_processing/plugin_event_multiplexer.py", line 259, in Reload
    Worker()
  File "~/miniconda3/lib/python3.6/site-packages/tensorboard/backend/event_processing/plugin_event_multiplexer.py", line 237, in Worker
    accumulator.Reload()
  File "~/miniconda3/lib/python3.6/site-packages/tensorboard/backend/event_processing/plugin_event_accumulator.py", line 183, in Reload
    for event in self._generator.Load():
  File "~/miniconda3/lib/python3.6/site-packages/tensorboard/backend/event_processing/directory_watcher.py", line 90, in Load
    for event in self._LoadInternal():
  File "~/miniconda3/lib/python3.6/site-packages/tensorboard/backend/event_processing/directory_watcher.py", line 120, in _LoadInternal
    for event in self._loader.Load():
  File "~/miniconda3/lib/python3.6/site-packages/tensorboard/backend/event_processing/event_file_loader.py", line 164, in Load
    for record in super(EventFileLoader, self).Load():
  File "~/miniconda3/lib/python3.6/site-packages/tensorboard/backend/event_processing/event_file_loader.py", line 139, in Load
    yield next(self._iterator)
  File "~/miniconda3/lib/python3.6/site-packages/tensorboard/backend/event_processing/event_file_loader.py", line 110, in __next__
    self._reader.GetNext()
  File "~/miniconda3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/pywrap_tensorflow.py", line 210, in GetNext
    header_str = self._read(8)
  File "~/miniconda3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/pywrap_tensorflow.py", line 274, in _read
    new_data = self.file_handle.read(n)
  File "~/miniconda3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/io/gfile.py", line 498, in read
    self.filename, self.binary_mode, read_size, self.continuation_token
  File "~/miniconda3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/io/gfile.py", line 290, in read
    stream = s3.Object(bucket, path).get(**args)["Body"].read()
  File "~/miniconda3/lib/python3.6/site-packages/boto3/resources/factory.py", line 520, in do_action
    response = action(self, *args, **kwargs)
  File "~/miniconda3/lib/python3.6/site-packages/boto3/resources/action.py", line 83, in __call__
    response = getattr(parent.meta.client, operation_name)(*args, **params)
  File "~/miniconda3/lib/python3.6/site-packages/botocore/client.py", line 316, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "~/miniconda3/lib/python3.6/site-packages/botocore/client.py", line 626, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidRange) when calling the GetObject operation: byte range bytes=40-16777256 for a file of length 40

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions