TensorBoard 1.7.0 is reserving GPU #1107
Description
TensorBoard version: 1.7.0 (upgrade from 1.6.0)
OS: Linux Ubuntu 16.04 (docker tensorflow/tensorflow:1.7.0-gpu-py3)
Python Version: 3.6.4
Issue
Since version 1.7.0, TensorBoard is occupying the GPU. This is not desired, as the GPUs are used for training. There exists CUDA_VISIBLE_DEVICES, but for backwards compatibility should be expected that this is not required. Also if the GPU is already used by training, tensorboard won't start, but use 100% CPU and you cannot cancel it by sigint.
Log Output
2018-04-06 14:58:04.297109: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2018-04-06 14:58:04.574507: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1344] Found device 0 with properties:
name: GeForce GTX TITAN X major: 5 minor: 2 memoryClockRate(GHz): 1.076
pciBusID: 0000:05:00.0
totalMemory: 11.92GiB freeMemory: 11.80GiB
2018-04-06 14:58:06.224336: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0