Documentation for torch.finfo doesn't match implementation #59526
Description
🐛 Bug
The output of torch.finfo(torch.float16) seems to be incorrect.
To Reproduce
Steps to reproduce the behavior:
>>> import torch
>>> torch.finfo(torch.float16)
finfo(resolution=0.001, min=-65504, max=65504, eps=0.000976562, tiny=6.10352e-05, dtype=float16)
>>> torch.tensor([1e-7], dtype=torch.float16)
tensor([1.1921e-07], dtype=torch.float16)
>>> torch.tensor([1.0], dtype=torch.float16) + torch.tensor([0.0006], dtype=torch.float16)
tensor([1.0010], dtype=torch.float16)
Expected behavior
Notice that 1e-7 is representable in fp16 but finfo indicates a tiny value much larger than that.
Also, 1.0 + 6e-4 != 1.0 but the reported eps is larger than 6e-4.
I expect tiny to equal the smallest positive value that's representable in fp16 and eps to equal the smallest representable value such that 1.0 + eps != 1.0.
Environment
PyTorch version: 1.8.0a0+7aef199
Is debug build: False
CUDA used to build PyTorch: 11.2
ROCM used to build PyTorch: N/A
OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.19.4
Libc version: glibc-2.27
Python version: 3.8 (64-bit runtime)
Python platform: Linux-4.15.0-118-generic-x86_64-with-glibc2.10
Is CUDA available: True
CUDA runtime version: 11.2.152
GPU models and configuration:
GPU 0: GeForce RTX 3090
GPU 1: GeForce GTX 1080 Ti
GPU 2: GeForce GTX 1080 Ti
GPU 3: GeForce GTX 1080 Ti
Nvidia driver version: 460.32.03
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.7.5.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn.so.8.1.0
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.1.0
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.1.0
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.1.0
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.1.0
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.1.0
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.1.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Versions of relevant libraries:
[pip3] numpy==1.19.5
[pip3] torch==1.8.0a0+7aef199
[pip3] torchaudio==0.7.0a0+a853dff
[pip3] torchvision==0.8.2
[conda] blas 1.0 mkl
[conda] cudatoolkit 10.2.89 hfd86e86_1
[conda] magma-cuda102 2.5.2 1 pytorch
[conda] mkl 2020.2 256
[conda] mkl-include 2020.2 256
[conda] mkl-service 2.3.0 py38he904b0f_0
[conda] mkl_fft 1.2.0 py38h23d657b_0
[conda] mkl_random 1.1.1 py38h0573a6f_0
[conda] numpy 1.19.5 pypi_0 pypi
[conda] torch 1.8.0a0+7aef199 pypi_0 pypi
[conda] torchaudio 0.7.0a0+a853dff pypi_0 pypi
[conda] torchvision 0.8.2 pypi_0 pypi