Nightly pip wheel+cu121 reports NCCL==2.18.1, but installs nvidia-nccl-cu12==2.19.3 #116977
Description
Command to install the latest nightly binary:
pip install --pre torch torchvision --index-url https://download.pytorch.org/whl/nightly/cu121
installs nvidia-nccl-cu12==2.19.3:
Looking in indexes: https://download.pytorch.org/whl/nightly/cu121
Collecting torch
Downloading https://download.pytorch.org/whl/nightly/cu121/torch-2.3.0.dev20240108%2Bcu121-cp310-cp310-linux_x86_64.whl (757.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 757.0/757.0 MB 36.2 MB/s eta 0:00:00
...
Collecting nvidia-nccl-cu12==2.19.3 (from torch)
Downloading https://download.pytorch.org/whl/nightly/cu121/nvidia_nccl_cu12-2.19.3-py3-none-manylinux1_x86_64.whl (166.0 MB)
...
However, PyTorch itself reports 2.18.1:
python -c "import torch; print(torch.__version__); print(torch.cuda.nccl.version())"
2.3.0.dev20240108+cu121
(2, 18, 1)I guess this version tag might be defined during the build process from e.g. here and then forwarded from here, but also source builds should use 2.19.3 based on the third_party/nccl tag.
Unsure, if this is just a reporting issue or if we are building the nightly binaries with NCCL==2.18.1 and executing them with 2.19.3, so creating this issue to dig a bit further into it.
cc @ezyang @gchanan @zou3519 @kadeng @seemethere @malfet @osalpekar @atalman