KEMBAR78
[ROCm] devtoolset / GCC11 upgrade on manylinux images - 1b of 2 (docker images) by pruthvistony · Pull Request #141609 · pytorch/pytorch · GitHub
Skip to content

Conversation

@pruthvistony
Copy link
Collaborator

@pruthvistony pruthvistony commented Nov 26, 2024

Upgrade gcc version from 9 to 11 on ROCm manylinux images.

Needed for #141423 since almalinux8-based manylinux2_28 images for ROCm (#140681) installs gcc-toolset-9, which installs gcc 9.2.1. However, PyTorch CMakeLists.txt enforces a minimum gcc version of 9.3.

cc @jeffdaily @sunway513 @jithunnair-amd @ROCmSupport @dllehr-amd @jataylo @hongxiayang @naromero77amd

@pytorch-bot
Copy link

pytorch-bot bot commented Nov 26, 2024

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/141609

Note: Links to docs will display an error until the docs builds have been completed.

❌ 5 New Failures, 1 Unrelated Failure

As of commit 22d1987 with merge base 8ba555e (image):

NEW FAILURES - The following jobs have failed:

FLAKY - The following job failed but was likely due to flakiness present on trunk:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@pytorch-bot pytorch-bot bot added ciflow/rocm Trigger "default" config CI on ROCm module: rocm AMD GPU support for Pytorch topic: not user facing topic category labels Nov 26, 2024
@pruthvistony pruthvistony added the rocm This tag is for PRs from ROCm team label Nov 26, 2024
@jithunnair-amd jithunnair-amd marked this pull request as draft November 27, 2024 05:02
@jithunnair-amd jithunnair-amd changed the title [ROCm] GCC upgrade on manylinux images [ROCm] devtoolset / GCC11 upgrade on manylinux images Nov 28, 2024
@jithunnair-amd jithunnair-amd changed the title [ROCm] devtoolset / GCC11 upgrade on manylinux images [ROCm] devtoolset / GCC11 upgrade on manylinux images - 1b of 2 (docker images) Nov 28, 2024
@jithunnair-amd
Copy link
Collaborator

The manylinux docker image build with LDFLAGS set to point to gcc-toolset-11 succeeded for 6.2.4 and 6.1.

The wheel builds using the above images also succeeded in PR 141423: eg. https://github.com/pytorch/pytorch/actions/runs/12068334721/job/33653245377

--   Shared LD flags       : -Wl,-rpath=/opt/rh/gcc-toolset-11/root/usr/lib64 -Wl,-rpath=/opt/rh/gcc-toolset-11/root/usr/lib -Wl,--no-as-needed -rdynamic
--   Static LD flags       :
--   Module LD flags       : -Wl,-rpath=/opt/rh/gcc-toolset-11/root/usr/lib64 -Wl,-rpath=/opt/rh/gcc-toolset-11/root/usr/lib

However, the wheel test jobs failed because they do not have AWS access set up, so we need to blind merge this PR to enable these changes to reflect in the dockerhub pytorch/manylinux2_28-builder:rocm6.2.4-main docker images, and hope that the wheel test jobs succeed.

@jithunnair-amd jithunnair-amd marked this pull request as ready for review November 28, 2024 17:13
@jithunnair-amd jithunnair-amd marked this pull request as draft November 28, 2024 17:18
@jithunnair-amd
Copy link
Collaborator

jithunnair-amd commented Nov 28, 2024

Just re-testing out two PyTorch build jobs that failed (in pull and rocm workflows) on the last round of CI runs (a bit surprising because the changes in this PR should not impact regular CI builds, but just want to be sure there's no negative impact of this PR on ROCm CI builds)...

@jithunnair-amd
Copy link
Collaborator

Just re-testing out two PyTorch build jobs that failed (in pull and rocm workflows) on the last round of CI runs (a bit surprising because the changes in this PR should not impact regular CI builds, but just want to be sure there's no negative impact of this PR on ROCm CI builds)...

PyTorch builds succeeded on retry: pull and rocm

@jithunnair-amd jithunnair-amd marked this pull request as ready for review November 28, 2024 19:08
@jithunnair-amd
Copy link
Collaborator

@pytorchbot merge -f "ROCm CI builds passed. ROCm test failures and other CI failures are unrelated"

@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

pytorchmergebot pushed a commit that referenced this pull request Dec 4, 2024
Depends on #140681 and #141609

Highlights:
* Upgrade binaries to ROCm6.2.4 to use latest docker images
* Remove pre-cxx11 builds for libtorch on ROCm
* Use manylinux2_28 docker images for ROCm
* Set `DESIRED_DEVTOOLSET=cxx-abi` (and hence `_GLIBCXX_USE_CXX11_ABI=1`) for ROCm manylinux2_28 wheels (ROCm RHEL8 packages also have GCC_ABI=1, so it keeps it consistent)

Pull Request resolved: #141423
Approved by: https://github.com/jeffdaily

Co-authored-by: Jeff Daily <jeff.daily@amd.com>
Co-authored-by: Pruthvi Madugundu <pruthvigithub@gmail.com>
pobin6 pushed a commit to pobin6/pytorch that referenced this pull request Dec 5, 2024
…er images) (pytorch#141609)

Upgrade gcc version from 9 to 11 on ROCm manylinux images.

Needed for pytorch#141423 since almalinux8-based manylinux2_28 images for ROCm (pytorch#140681) installs gcc-toolset-9, which installs [gcc 9.2.1](https://pkgs.org/download/gcc-toolset-9-gcc-c++). However, PyTorch CMakeLists.txt enforces a [minimum gcc version of 9.3](https://github.com/pytorch/pytorch/blob/5318bf8baf19fecda365c185cd81196e3cfb08e3/CMakeLists.txt#L61).

Pull Request resolved: pytorch#141609
Approved by: https://github.com/jeffdaily

Co-authored-by: Jithun Nair <jithun.nair@amd.com>
pobin6 pushed a commit to pobin6/pytorch that referenced this pull request Dec 5, 2024
)

Depends on pytorch#140681 and pytorch#141609

Highlights:
* Upgrade binaries to ROCm6.2.4 to use latest docker images
* Remove pre-cxx11 builds for libtorch on ROCm
* Use manylinux2_28 docker images for ROCm
* Set `DESIRED_DEVTOOLSET=cxx-abi` (and hence `_GLIBCXX_USE_CXX11_ABI=1`) for ROCm manylinux2_28 wheels (ROCm RHEL8 packages also have GCC_ABI=1, so it keeps it consistent)

Pull Request resolved: pytorch#141423
Approved by: https://github.com/jeffdaily

Co-authored-by: Jeff Daily <jeff.daily@amd.com>
Co-authored-by: Pruthvi Madugundu <pruthvigithub@gmail.com>
AmdSampsa pushed a commit to AmdSampsa/pytorch that referenced this pull request Dec 9, 2024
)

Depends on pytorch#140681 and pytorch#141609

Highlights:
* Upgrade binaries to ROCm6.2.4 to use latest docker images
* Remove pre-cxx11 builds for libtorch on ROCm
* Use manylinux2_28 docker images for ROCm
* Set `DESIRED_DEVTOOLSET=cxx-abi` (and hence `_GLIBCXX_USE_CXX11_ABI=1`) for ROCm manylinux2_28 wheels (ROCm RHEL8 packages also have GCC_ABI=1, so it keeps it consistent)

Pull Request resolved: pytorch#141423
Approved by: https://github.com/jeffdaily

Co-authored-by: Jeff Daily <jeff.daily@amd.com>
Co-authored-by: Pruthvi Madugundu <pruthvigithub@gmail.com>
jithunnair-amd added a commit to ROCm/builder that referenced this pull request Jun 13, 2025
### Motivation: 
* Use newer devtoolset11 to sync with upstream:
pytorch/pytorch#141609

### Changes:
* Use devtoolset 11
* Set LDFLAGS and DEVTOOLSET_VERSION
(cherry picked from commit a80f1ef)

Relates to: ROCm/rocAutomation#937

Validation: 
* 2.7:
http://rocm-ci.amd.com/job/pytorch-pipeline-manylinux-wheel-builder_rel-6.4/379/
* 2.5:
http://rocm-ci.amd.com/job/pytorch-pipeline-manylinux-wheel-builder_rel-6.4/400/
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ciflow/rocm Trigger "default" config CI on ROCm Merged module: rocm AMD GPU support for Pytorch open source rocm This tag is for PRs from ROCm team topic: not user facing topic category

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants