[ROCm][CI] upgrade to 6.4.2 patch release #158887

naromero77amd · 2025-07-22T23:32:03Z

Upgrade to ROCm 6.4.2.

cc @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang

pytorch-bot · 2025-07-22T23:32:06Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/158887

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 11 New Failures, 1 Cancelled Job, 22 Unrelated Failures

As of commit faa0de7 with merge base ee97dbf ():

NEW FAILURES - The following jobs have failed:

periodic / linux-jammy-cuda12.8-py3.10-gcc11 / test (nogpu_NO_AVX2, 1, 2, lf.linux.4xlarge) (gh)
RuntimeError:
s390x-periodic / linux-manylinux-2_28-py3-cpu-s390x / test (default, 1, 10, linux.s390x) (gh)
Process completed with exit code 1.
s390x-periodic / linux-manylinux-2_28-py3-cpu-s390x / test (default, 10, 10, linux.s390x) (gh)
Process completed with exit code 1.
s390x-periodic / linux-manylinux-2_28-py3-cpu-s390x / test (default, 2, 10, linux.s390x) (gh)
Process completed with exit code 1.
s390x-periodic / linux-manylinux-2_28-py3-cpu-s390x / test (default, 3, 10, linux.s390x) (gh)
Process completed with exit code 1.
s390x-periodic / linux-manylinux-2_28-py3-cpu-s390x / test (default, 4, 10, linux.s390x) (gh)
Process completed with exit code 1.
s390x-periodic / linux-manylinux-2_28-py3-cpu-s390x / test (default, 5, 10, linux.s390x) (gh)
Process completed with exit code 1.
s390x-periodic / linux-manylinux-2_28-py3-cpu-s390x / test (default, 6, 10, linux.s390x) (gh)
Process completed with exit code 1.
s390x-periodic / linux-manylinux-2_28-py3-cpu-s390x / test (default, 7, 10, linux.s390x) (gh)
Process completed with exit code 1.
s390x-periodic / linux-manylinux-2_28-py3-cpu-s390x / test (default, 8, 10, linux.s390x) (gh)
Process completed with exit code 1.
s390x-periodic / linux-manylinux-2_28-py3-cpu-s390x / test (default, 9, 10, linux.s390x) (gh)
Process completed with exit code 1.

CANCELLED JOB - The following job was cancelled. Please retry:

Check Labels (gh)

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

inductor-periodic / cuda12.8-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) (gh) (similar failure)
functorch_dp_cifar10
periodic / linux-jammy-cuda12.4-py3.10-gcc11 / test (legacy_nvidia_driver, 1, 5, lf.linux.4xlarge.nvidia.gpu) (gh) (disabled by #134745, #134746)
test_cuda_expandable_segments.py::TestCuda::test_record_stream
periodic / linux-jammy-cuda12.4-py3.10-gcc11-sm89 / test (default, 1, 5, lf.linux.g6.4xlarge.experimental.nvidia.gpu) (gh) (disabled by #156691)
dynamo/test_repros.py::ReproTests::test_relative_import_no_modulename
periodic / linux-jammy-cuda12.8-py3-gcc11-slow-gradcheck / test (default, 1, 8, lf.linux.g5.4xlarge.nvidia.gpu, module:slowgradcheck) (gh) (disabled by #156755)
inductor/test_cudagraph_trees.py::CudaGraphTreeTests::test_empty_storage
pull / linux-jammy-cuda12.8-py3.10-gcc11-sm89 / test (default, 1, 5, linux.g6.4xlarge.experimental.nvidia.gpu) (gh) (disabled by #156755)
inductor/test_cudagraph_trees.py::CudaGraphTreeTests::test_empty_storage

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

inductor-periodic / cuda12.8-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (aot_eager_timm, 2, 2, linux.g5.4xlarge.nvidia.gpu) (gh) (trunk failure)
mobilenetv3_large_100
inductor-periodic / cuda12.8-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_timm, 2, 2, linux.g5.4xlarge.nvidia.gpu) (gh) (trunk failure)
mobilenetv3_large_100
inductor-periodic / linux-jammy-cpu-py3.9-gcc11-inductor / test (cpu_inductor_amp_freezing_torchbench, 1, 2, linux.8xlarge.amx) (gh) (trunk failure)
detectron2_fcos_r_50_fpn
inductor-periodic / linux-jammy-cpu-py3.9-gcc11-inductor / test (cpu_inductor_amp_freezing_torchbench, 2, 2, linux.8xlarge.amx) (gh) (trunk failure)
vision_maskrcnn
inductor-periodic / linux-jammy-cpu-py3.9-gcc11-inductor / test (dynamic_cpu_aot_inductor_freezing_torchbench, 1, 2, linux.8xlarge.amx) (gh) (trunk failure)
basic_gnn_sage
periodic / linux-jammy-cuda12.8-py3.10-gcc11 / test (nogpu_AVX512, 1, 3, lf.linux.4xlarge) (gh) (trunk failure)
Process completed with exit code 2.
periodic / linux-jammy-cuda12.8-py3.10-gcc9-debug / test (default, 1, 7, lf.linux.4xlarge.nvidia.gpu, oncall:debug-build) (gh) (trunk failure)
RuntimeError: THPUtils_unpackInt(arg) != 424242 INTERNAL ASSERT FAILED at "/var/lib/jenkins/workspace/torch/csrc/Module.cpp":315, please report a bug to PyTorch. Expect anything but 424242 as an input for debug builds
pull / linux-jammy-cuda12.8-py3.10-gcc11 / test (default, 1, 5, linux.4xlarge.nvidia.gpu) (gh) (trunk failure)
Process completed with exit code 2.
pull / linux-jammy-py3.10-clang18-asan / test (default, 1, 6, linux.4xlarge) (gh) (trunk failure)
Process completed with exit code 127.
pull / linux-jammy-py3.13-clang12 / test (crossref, 1, 2, linux.2xlarge) (gh) (trunk failure)
Process completed with exit code 2.
pull / linux-jammy-py3.13-clang12 / test (default, 1, 5, linux.4xlarge) (gh) (trunk failure)
Process completed with exit code 2.
pull / linux-jammy-py3.9-clang12 / test (crossref, 1, 2, linux.2xlarge) (gh) (trunk failure)
Process completed with exit code 2.
pull / linux-jammy-py3.9-clang12 / test (default, 1, 5, linux.4xlarge) (gh) (trunk failure)
Process completed with exit code 2.
pull / linux-jammy-py3.9-gcc11 / test (default, 1, 5, linux.2xlarge) (gh) (trunk failure)
Process completed with exit code 2.
rocm / linux-jammy-rocm-py3.10 / test (default, 1, 6, linux.rocm.gpu.2) (gh) (trunk failure)
Process completed with exit code 2.
rocm-mi300 / linux-noble-rocm-py3.12-mi300 / test (default, 1, 6, linux.rocm.gpu.mi300.2) (gh) (trunk failure)
Process completed with exit code 2.

UNSTABLE - The following job is marked as unstable, possibly due to flakiness on trunk:

pull / linux-jammy-py3_9-clang9-xla / test (xla, 1, 1, linux.12xlarge, unstable) (gh) (#158876)
sccache: error: couldn't connect to server

This comment was automatically generated by Dr. CI and updates every 15 minutes.

jeffdaily · 2025-07-25T03:43:34Z

@pytorchbot merge -f "rocm-only, full suite of ROCm tests are passing"

pytorchmergebot · 2025-07-25T03:45:12Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

Upgrade to ROCm 6.4.2. Pull Request resolved: #158887 Approved by: https://github.com/jeffdaily Co-authored-by: Jeff Daily <jeff.daily@amd.com>

This reverts commit 16c0ccd.

@jeffdaily

This is a partial cherry-pick from upstream: pytorch#158887 The TunableOp UT appears to fail with different versions of ROCm, but the real issue is the that regex in the UT is incorrect. @jeffdaily discovered this when we upgraded the nightly wheels to ROCm 6.4.2 in upstream PyTorch. Will resolve SWDEV-548689.

@jeffdaily

This is a partial cherry-pick from upstream: pytorch#158887 The TunableOp UT appears to fail with different versions of ROCm, but the real issue is the that regex in the UT is incorrect. @jeffdaily discovered this when we upgraded the nightly wheels to ROCm 6.4.2 in upstream PyTorch. Will resolve SWDEV-548689.

[ROCm][CI] upgrade to 6.4.2 patch release

d807a37

naromero77amd requested a review from jeffdaily as a code owner July 22, 2025 23:32

pytorch-bot bot added module: rocm AMD GPU support for Pytorch topic: not user facing topic category labels Jul 22, 2025

pytorchbot added the open source label Jul 22, 2025

soulitzer added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Jul 23, 2025

pytorch-bot bot added the ciflow/rocm Trigger "default" config CI on ROCm label Jul 23, 2025

naromero77amd mentioned this pull request Jul 23, 2025

[ROCm][tunableop] Validator UT requires fix to regex #158982

Closed

Fix regex in validator unit test.

6250dcb

naromero77amd requested review from IvanYashchuk, lezcano and nikitaved as code owners July 23, 2025 21:51

jeffdaily added the ciflow/periodic Trigger jobs ran periodically on master (periodic.yml) on the PR label Jul 23, 2025

jeffdaily approved these changes Jul 24, 2025

View reviewed changes

pytorchmergebot added the merging label Jul 25, 2025

pytorchmergebot closed this in 16c0ccd Jul 25, 2025

pytorchmergebot added Merged and removed merging labels Jul 25, 2025

yangw-dev pushed a commit that referenced this pull request Aug 1, 2025

[ROCm][CI] upgrade to 6.4.2 patch release (#158887)

9c21c92

Upgrade to ROCm 6.4.2. Pull Request resolved: #158887 Approved by: https://github.com/jeffdaily Co-authored-by: Jeff Daily <jeff.daily@amd.com>

jataylo added a commit to jataylo/pytorch that referenced this pull request Aug 7, 2025

Revert "[ROCm][CI] upgrade to 6.4.2 patch release (pytorch#158887)"

a35507c

This reverts commit 16c0ccd.

pytorchmergebot pushed a commit to jataylo/pytorch that referenced this pull request Aug 8, 2025

Revert "[ROCm][CI] upgrade to 6.4.2 patch release (pytorch#158887)"

679de30

This reverts commit 16c0ccd.

pytorch-bot bot pushed a commit that referenced this pull request Aug 9, 2025

Revert "[ROCm][CI] upgrade to 6.4.2 patch release (#158887)"

9817ce6

This reverts commit 16c0ccd.

pytorchmergebot pushed a commit to jataylo/pytorch that referenced this pull request Aug 10, 2025

Revert "[ROCm][CI] upgrade to 6.4.2 patch release (pytorch#158887)"

b97ad23

This reverts commit 16c0ccd.

naromero77amd mentioned this pull request Aug 25, 2025

[release/2.7][ROCm] Bug fix TunableOp validator UT. ROCm/pytorch#2576

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[ROCm][CI] upgrade to 6.4.2 patch release #158887

[ROCm][CI] upgrade to 6.4.2 patch release #158887

Uh oh!

naromero77amd commented Jul 22, 2025 •

edited by pytorch-bot bot

Loading

Uh oh!

pytorch-bot bot commented Jul 22, 2025 •

edited

Loading

Uh oh!

jeffdaily commented Jul 25, 2025

Uh oh!

pytorchmergebot commented Jul 25, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

[ROCm][CI] upgrade to 6.4.2 patch release #158887

[ROCm][CI] upgrade to 6.4.2 patch release #158887

Uh oh!

Conversation

naromero77amd commented Jul 22, 2025 • edited by pytorch-bot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/158887

❌ 11 New Failures, 1 Cancelled Job, 22 Unrelated Failures

Uh oh!

jeffdaily commented Jul 25, 2025

Uh oh!

pytorchmergebot commented Jul 25, 2025

Merge started

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

naromero77amd commented Jul 22, 2025 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Jul 22, 2025 •

edited

Loading