KEMBAR78
[ROCm CI] Migrate to MI325 Capacity. by saienduri · Pull Request #159059 · pytorch/pytorch · GitHub
Skip to content

[ROCm CI] Migrate to MI325 Capacity. #159059

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 33 commits into from
Closed

[ROCm CI] Migrate to MI325 Capacity. #159059

wants to merge 33 commits into from

Conversation

@saienduri
Copy link
Collaborator

@saienduri saienduri commented Jul 24, 2025 •
edited
Loading

This PR moves PyTorch CI capacity from mi300 to a new, larger mi325 cluster. Both of these GPUs are the same architecture gfx942 and our testing plans don't change within an architecture, so we pool them under the same label linux.rocm.gpu.gfx942.<#gpus> with this PR as well to reduce overhead and confusion.

cc @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang @naromero77amd @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @chenyang78 @kadeng @muchulee8 @amjames @chauhang @aakhundov @coconutruben

@saienduri saienduri requested a review from a team as a code owner July 24, 2025 17:25
@pytorch-bot
Copy link

pytorch-bot bot commented Jul 24, 2025 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/159059

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (2 Unrelated Failures)

As of commit f430e83 with merge base 67e68e0 (image):

UNSTABLE - The following jobs are marked as unstable, possibly due to flakiness on trunk:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@linux-foundation-easycla
Copy link

linux-foundation-easycla bot commented Jul 24, 2025 •
edited
Loading

CLA Signed

The committers listed above are authorized under a signed CLA.

@pytorch-bot pytorch-bot bot added the topic: not user facing topic category label Jul 24, 2025
@facebook-github-bot facebook-github-bot added the module: rocm AMD GPU support for Pytorch label Jul 24, 2025
@saienduri saienduri mentioned this pull request Jul 24, 2025
Closed
@saienduri saienduri added ciflow/rocm-mi300 Trigger "default" config CI on ROCm MI300 ciflow/periodic-rocm-mi300 Trigger "distributed" config CI on ROCm MI300 ciflow/inductor-rocm Trigger "inductor" config CI on ROCm labels Jul 24, 2025
@pytorch-bot pytorch-bot bot removed ciflow/inductor-rocm Trigger "inductor" config CI on ROCm ciflow/rocm-mi300 Trigger "default" config CI on ROCm MI300 ciflow/periodic-rocm-mi300 Trigger "distributed" config CI on ROCm MI300 labels Jul 25, 2025
@saienduri saienduri added ciflow/inductor-rocm Trigger "inductor" config CI on ROCm ciflow/rocm-mi300 Trigger "default" config CI on ROCm MI300 ciflow/periodic-rocm-mi300 Trigger "distributed" config CI on ROCm MI300 labels Jul 25, 2025
@pytorch-bot pytorch-bot bot removed the ciflow/inductor-rocm Trigger "inductor" config CI on ROCm label Jul 25, 2025
@pytorch-bot pytorch-bot bot removed ciflow/inductor ciflow/rocm Trigger "default" config CI on ROCm ciflow/inductor-rocm Trigger "inductor" config CI on ROCm ciflow/rocm-mi300 Trigger "default" config CI on ROCm MI300 ciflow/periodic-rocm-mi300 Trigger "distributed" config CI on ROCm MI300 labels Jul 29, 2025
@saienduri saienduri added ciflow/inductor-rocm Trigger "inductor" config CI on ROCm ciflow/rocm-mi300 Trigger "default" config CI on ROCm MI300 ciflow/periodic-rocm-mi300 Trigger "distributed" config CI on ROCm MI300 labels Jul 29, 2025
@saienduri saienduri requested a review from huydhn July 30, 2025 00:08
@atalman
Copy link
Contributor

atalman commented Jul 30, 2025

@pytorchmergebot merge -i

@pytorch-bot pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Jul 30, 2025
@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged while ignoring the following 2 checks: pull / linux-jammy-py3_9-clang9-xla / test (xla, 1, 1, lf.linux.12xlarge, unstable), rocm-mi300 / linux-noble-rocm-py3.12-mi300 / test (default, 5, 6, linux.rocm.gpu.gfx942.2, unstable)

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@jithunnair-amd
Copy link
Collaborator

@pytorchbot merge -f "Force merging since this is time-sensitive (losing MI300 capacity today EOD), and we already verified the full ROCm workflows"

@pytorchmergebot
Copy link
Collaborator

The merge job was canceled or timed out. This most often happen if two merge requests were issued for the same PR, or if merge job was waiting for more than 6 hours for tests to finish. In later case, please do not hesitate to reissue the merge command
For more information see pytorch-bot wiki.

@jithunnair-amd
Copy link
Collaborator

@pytorchbot merge -f "Force merging since this is time-sensitive (losing MI300 capacity today EOD), and we already verified the full ROCm workflows"

@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

yangw-dev pushed a commit that referenced this pull request Aug 1, 2025
@saienduri @deedongala @yangw-dev
[ROCm CI] Migrate to MI325 Capacity. (#159059)
161c92c 
This PR moves PyTorch CI capacity from mi300 to a new, larger mi325 cluster. Both of these GPUs are the same architecture gfx942 and our testing plans don't change within an architecture, so we pool them under the same label `linux.rocm.gpu.gfx942.<#gpus>` with this PR as well to reduce overhead and confusion.

Pull Request resolved: #159059
Approved by: https://github.com/jithunnair-amd, https://github.com/atalman

Co-authored-by: deedongala <deekshitha.dongala@amd.com>
This was referenced Aug 1, 2025
Merged
Closed
Merged
pytorchmergebot pushed a commit that referenced this pull request Aug 4, 2025
@amdfaa @pytorchmergebot
[ROCm CI] Migrate to MI325 Capacity (#159649)
1d3eef2 
Migrate mi300s to gfx942.

Related to #159059

Pull Request resolved: #159649
Approved by: https://github.com/huydhn
markc-614 pushed a commit to markc-614/pytorch that referenced this pull request Sep 17, 2025
@amdfaa @markc-614
[ROCm CI] Migrate to MI325 Capacity (pytorch#159649)
cb2b7a9 
Migrate mi300s to gfx942.

Related to pytorch#159059

Pull Request resolved: pytorch#159649
Approved by: https://github.com/huydhn
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@atalman atalman atalman approved these changes

@jithunnair-amd jithunnair-amd jithunnair-amd approved these changes

@malfet malfet Awaiting requested review from malfet

@seemethere seemethere Awaiting requested review from seemethere

@huydhn huydhn Awaiting requested review from huydhn

Assignees

No one assigned

Labels

ci-no-td Do not run TD on this PR ciflow/inductor-rocm Trigger "inductor" config CI on ROCm ciflow/periodic-rocm-mi300 Trigger "distributed" config CI on ROCm MI300 ciflow/rocm-mi300 Trigger "default" config CI on ROCm MI300 ciflow/trunk Trigger trunk jobs on your pull request keep-going Don't stop on first failure, keep running tests until the end Merged module: inductor module: rocm AMD GPU support for Pytorch open source topic: not user facing topic category triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@saienduri @jithunnair-amd @atalman @pytorchmergebot @facebook-github-bot @pytorchbot @HDCharles @deedongala