[ROCm] Enable USE_FBGEMM_GENAI #160676

cthi · 2025-08-14T20:39:07Z

X-link: https://github.com/facebookresearch/FBGEMM/pull/1728

In this diff we enable the support for the new FBGEMM backed FP8 _scaled_grouped_mm on ROCm. For now we only enable support for gfx942 as that is what we have thoroughly tested performance and correctness on.

Rollback Plan:

Differential Revision: D79564024

Test Plan:

Ensure builds with:

USE_FBGEMM_GENAI=1 and without gfx942
USE_FBGEMM_GENAI=1 and with gfx942
USE_FBGEMM_GENAI=1 and all current PYTORCH_ROCM_ARCH

cc @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang @naromero77amd

pytorch-bot · 2025-08-14T20:39:10Z

This appears to be a diff that was exported from phabricator, but the PR author does not have sufficient permissions to run CI. @cthi, please do step 2 of internal wiki to get write access so you do not need to get CI approvals in the future. If you think this is a mistake, please contact the Pytorch Dev Infra team.

pytorch-bot · 2025-08-14T20:39:13Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/160676

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (1 Unrelated Failure)

As of commit 68b8727 with merge base bc4db2c ():

UNSTABLE - The following job is marked as unstable, possibly due to flakiness on trunk:

rocm / linux-jammy-rocm-py3.10 / test (default, 4, 6, linux.rocm.gpu.2) (gh) (#156098)
##[error]The operation was canceled.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

facebook-github-bot · 2025-08-14T20:39:17Z

This pull request was exported from Phabricator. Differential Revision: D79564024

pytorch-bot · 2025-08-14T20:39:21Z

To add the ciflow label ciflow/rocm please first approve the workflows that are awaiting approval (scroll to the bottom of this page).

This helps ensure we don't trigger CI on this PR until it is actually authorized to do so. Please ping one of the reviewers if you do not have access to approve and run workflows.

Summary: X-link: pytorch/FBGEMM#4703 X-link: facebookresearch/FBGEMM#1728 In this diff we enable the support for the new FBGEMM backed FP8 _scaled_grouped_mm on ROCm. For now we only enable support for gfx942 as that is what we have thoroughly tested performance and correctness on. Test Plan: Will ensure CI is green internally. Ensure the op can be called, added it into the fbgemm testing script: ``` HIP_VISIBLE_DEVICES=1 buck2 run @//mode/{opt,amd-gpu,inplace} -c fbcode.enable_gpu_sections=true -c fbcode.triton_backend=amd -c fbcode.rocm_arch=mi300 //deeplearning/fbgemm/fbgemm_gpu/experimental/gen_ai/bench:quantize_bench -- --kernels=ck_rowwise_grouped,ck_grouped_stacked_torch_2d3d,scaled_grouped_mm_rowwise --grouped --M=128 --N=2048 --K=5120 --groups=16 ``` ``` ck_rowwise_grouped sim: 115.344. ck_rowwise_grouped ms: 0.167. ck_rowwise_grouped TFLOPS: 257.254. ck_rowwise_grouped GB/s: 1117.952. Average metrics over 1 iterations: ck_grouped_stacked_torch_2d3d sim: 115.344. ck_grouped_stacked_torch_2d3d ms: 0.072. ck_grouped_stacked_torch_2d3d TFLOPS: 594.511. ck_grouped_stacked_torch_2d3d GB/s: 2583.570. Average metrics over 1 iterations: scaled_grouped_mm_rowwise sim: 115.344. scaled_grouped_mm_rowwise ms: 0.074. scaled_grouped_mm_rowwise TFLOPS: 576.926. scaled_grouped_mm_rowwise GB/s: 2507.148. ``` Rollback Plan: Differential Revision: D79564024

facebook-github-bot · 2025-08-14T20:54:58Z

This pull request was exported from Phabricator. Differential Revision: D79564024

Summary: X-link: pytorch/pytorch#160676 X-link: facebookresearch/FBGEMM#1728 In this diff we enable the support for the new FBGEMM backed FP8 _scaled_grouped_mm on ROCm. For now we only enable support for gfx942 as that is what we have thoroughly tested performance and correctness on. Differential Revision: D79564024

Summary: X-link: pytorch/FBGEMM#4703 X-link: facebookresearch/FBGEMM#1728 In this diff we enable the support for the new FBGEMM backed FP8 _scaled_grouped_mm on ROCm. For now we only enable support for gfx942 as that is what we have thoroughly tested performance and correctness on. Test Plan: Will ensure CI is green internally. Ensure the op can be called, added it into the fbgemm testing script: ``` HIP_VISIBLE_DEVICES=1 buck2 run @//mode/{opt,amd-gpu,inplace} -c fbcode.enable_gpu_sections=true -c fbcode.triton_backend=amd -c fbcode.rocm_arch=mi300 //deeplearning/fbgemm/fbgemm_gpu/experimental/gen_ai/bench:quantize_bench -- --kernels=ck_rowwise_grouped,ck_grouped_stacked_torch_2d3d,scaled_grouped_mm_rowwise --grouped --M=128 --N=2048 --K=5120 --groups=16 ``` ``` ck_rowwise_grouped sim: 115.344. ck_rowwise_grouped ms: 0.167. ck_rowwise_grouped TFLOPS: 257.254. ck_rowwise_grouped GB/s: 1117.952. Average metrics over 1 iterations: ck_grouped_stacked_torch_2d3d sim: 115.344. ck_grouped_stacked_torch_2d3d ms: 0.072. ck_grouped_stacked_torch_2d3d TFLOPS: 594.511. ck_grouped_stacked_torch_2d3d GB/s: 2583.570. Average metrics over 1 iterations: scaled_grouped_mm_rowwise sim: 115.344. scaled_grouped_mm_rowwise ms: 0.074. scaled_grouped_mm_rowwise TFLOPS: 576.926. scaled_grouped_mm_rowwise GB/s: 2507.148. ``` Rollback Plan: Differential Revision: D79564024

facebook-github-bot · 2025-08-25T21:34:39Z

This pull request was exported from Phabricator. Differential Revision: D79564024

Summary: X-link: pytorch/pytorch#160676 X-link: facebookresearch/FBGEMM#1728 In this diff we enable the support for the new FBGEMM backed FP8 _scaled_grouped_mm on ROCm. For now we only enable support for gfx942 as that is what we have thoroughly tested performance and correctness on. Differential Revision: D79564024

facebook-github-bot · 2025-08-26T14:29:07Z

This pull request was exported from Phabricator. Differential Revision: D79564024

Summary: X-link: pytorch/FBGEMM#4703 X-link: facebookresearch/FBGEMM#1728 In this diff we enable the support for the new FBGEMM backed FP8 `torch._scaled_grouped_mm` on ROCm. For now we only enable support for `gfx942` as that is what we have thoroughly tested performance and correctness on. Test Plan: Will ensure CI is green internally. Ensure the op can be called, added it into the fbgemm testing script: ``` HIP_VISIBLE_DEVICES=1 buck2 run @//mode/{opt,amd-gpu,inplace} -c fbcode.enable_gpu_sections=true -c fbcode.triton_backend=amd -c fbcode.rocm_arch=mi300 //deeplearning/fbgemm/fbgemm_gpu/experimental/gen_ai/bench:quantize_bench -- --kernels=ck_rowwise_grouped,ck_grouped_stacked_torch_2d3d,scaled_grouped_mm_rowwise --grouped --M=128 --N=2048 --K=5120 --groups=16 ``` ``` ck_rowwise_grouped sim: 115.344. ck_rowwise_grouped ms: 0.167. ck_rowwise_grouped TFLOPS: 257.254. ck_rowwise_grouped GB/s: 1117.952. Average metrics over 1 iterations: ck_grouped_stacked_torch_2d3d sim: 115.344. ck_grouped_stacked_torch_2d3d ms: 0.072. ck_grouped_stacked_torch_2d3d TFLOPS: 594.511. ck_grouped_stacked_torch_2d3d GB/s: 2583.570. Average metrics over 1 iterations: scaled_grouped_mm_rowwise sim: 115.344. scaled_grouped_mm_rowwise ms: 0.074. scaled_grouped_mm_rowwise TFLOPS: 576.926. scaled_grouped_mm_rowwise GB/s: 2507.148. ``` Rollback Plan: Differential Revision: D79564024

facebook-github-bot · 2025-09-02T14:36:27Z

This pull request was exported from Phabricator. Differential Revision: D79564024

drisspg · 2025-09-03T19:43:51Z

aten/src/ATen/CMakeLists.txt

+    # This is rather hacky, I could not figure out a clean solution :(
+    set(HIP_CLANG_FLAGS_ORIGINAL ${HIP_CLANG_FLAGS})
+    string(REGEX REPLACE "--offload-arch=[^ ]*" "" FILTERED_HIP_CLANG_FLAGS "${HIP_CLANG_FLAGS}")
+    list(APPEND FILTERED_HIP_CLANG_FLAGS --offload-arch=gfx942;)


looks pretty similar to

pytorch/cmake/Codegen.cmake

Line 91 in 01f66d0

torch_cuda_get_nvcc_gencode_flag(_existing_arch_flags)

Summary: X-link: pytorch/pytorch#160676 Pull Request resolved: #4703 X-link: facebookresearch/FBGEMM#1728 In this diff we enable the support for the new FBGEMM backed FP8 `torch._scaled_grouped_mm` on ROCm. For now we only enable support for `gfx942` as that is what we have thoroughly tested performance and correctness on. Reviewed By: drisspg Differential Revision: D79564024 fbshipit-source-id: bf2aa1a3eee43d0e47e9ba1e5514152e502da35f

facebook-github-bot · 2025-09-04T07:04:20Z

@pytorchbot merge

(Initiating merge automatically since Phabricator Diff has merged)

pytorchmergebot · 2025-09-04T07:07:34Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

This reverts commit 69a25f6.

Summary: X-link: pytorch/FBGEMM#4703 X-link: https://github.com/facebookresearch/FBGEMM/pull/1728 In this diff we enable the support for the new FBGEMM backed FP8 _scaled_grouped_mm on ROCm. For now we only enable support for `gfx942` as that is what we have thoroughly tested performance and correctness on. Rollback Plan: Differential Revision: D79564024 Test Plan: Ensure builds with: - `USE_FBGEMM_GENAI=1` and without gfx942 - `USE_FBGEMM_GENAI=1` and with gfx942 - `USE_FBGEMM_GENAI=1` and all current [`PYTORCH_ROCM_ARCH`](https://github.com/pytorch/pytorch/blob/9491d289b329e4ba4a9f5f5b1be7960671bb7840/.ci/docker/libtorch/build.sh#L48) Pull Request resolved: pytorch#160676 Approved by: https://github.com/drisspg

pytorch-bot bot added module: rocm AMD GPU support for Pytorch topic: not user facing topic category labels Aug 14, 2025

pytorch-bot bot added the ciflow/rocm Trigger "default" config CI on ROCm label Aug 14, 2025

facebook-github-bot added the fb-exported label Aug 14, 2025

pytorch-bot bot added ciflow/rocm Trigger "default" config CI on ROCm and removed ciflow/rocm Trigger "default" config CI on ROCm labels Aug 14, 2025

cthi force-pushed the export-D79564024 branch from dfecffb to 2775878 Compare August 14, 2025 20:54

pytorch-bot bot removed the ciflow/rocm Trigger "default" config CI on ROCm label Aug 14, 2025

drisspg added the ciflow/rocm Trigger "default" config CI on ROCm label Aug 14, 2025

cthi force-pushed the export-D79564024 branch from 2775878 to 4a582fe Compare August 25, 2025 21:34

cthi force-pushed the export-D79564024 branch from 4a582fe to 77d547a Compare August 26, 2025 14:28

cthi force-pushed the export-D79564024 branch from 77d547a to 68b8727 Compare September 2, 2025 14:36

cthi requested a review from drisspg September 3, 2025 13:34

cthi changed the title ~~[WIP][ROCm] Enable USE_FBGEMM_GENAI~~ [ROCm] Enable USE_FBGEMM_GENAI Sep 3, 2025

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Sep 3, 2025

drisspg reviewed Sep 3, 2025

View reviewed changes

drisspg approved these changes Sep 3, 2025

View reviewed changes

pytorchmergebot added the merging label Sep 4, 2025

pytorchmergebot closed this in 69a25f6 Sep 4, 2025

pytorchmergebot added Merged and removed merging labels Sep 4, 2025

jagadish-amd mentioned this pull request Sep 6, 2025

[ROCm] FP8 _scaled_grouped_mm & BF16 _grouped_mm support #161366

Open

pragupta added a commit to ROCm/pytorch that referenced this pull request Sep 9, 2025

Revert "[ROCm] Enable USE_FBGEMM_GENAI (pytorch#160676)"

304889c

This reverts commit 69a25f6.

pragupta added a commit to ROCm/pytorch that referenced this pull request Sep 10, 2025

Revert "[ROCm] Enable USE_FBGEMM_GENAI (pytorch#160676)"

dba8539

This reverts commit 69a25f6.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[ROCm] Enable USE_FBGEMM_GENAI #160676

[ROCm] Enable USE_FBGEMM_GENAI #160676

cthi commented Aug 14, 2025 •

edited

Loading

Uh oh!

pytorch-bot bot commented Aug 14, 2025

Uh oh!

pytorch-bot bot commented Aug 14, 2025 •

edited

Loading

Uh oh!

facebook-github-bot commented Aug 14, 2025

Uh oh!

pytorch-bot bot commented Aug 14, 2025

Uh oh!

facebook-github-bot commented Aug 14, 2025

Uh oh!

facebook-github-bot commented Aug 25, 2025

Uh oh!

facebook-github-bot commented Aug 26, 2025

Uh oh!

facebook-github-bot commented Sep 2, 2025

Uh oh!

drisspg Sep 3, 2025

Uh oh!

facebook-github-bot commented Sep 4, 2025

Uh oh!

pytorchmergebot commented Sep 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

[ROCm] Enable USE_FBGEMM_GENAI #160676

[ROCm] Enable USE_FBGEMM_GENAI #160676

Conversation

cthi commented Aug 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Aug 14, 2025

Uh oh!

pytorch-bot bot commented Aug 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/160676

✅ You can merge normally! (1 Unrelated Failure)

Uh oh!

facebook-github-bot commented Aug 14, 2025

Uh oh!

pytorch-bot bot commented Aug 14, 2025

Uh oh!

facebook-github-bot commented Aug 14, 2025

Uh oh!

facebook-github-bot commented Aug 25, 2025

Uh oh!

facebook-github-bot commented Aug 26, 2025

Uh oh!

facebook-github-bot commented Sep 2, 2025

Uh oh!

drisspg Sep 3, 2025

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot commented Sep 4, 2025

Uh oh!

pytorchmergebot commented Sep 4, 2025

Merge started

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

cthi commented Aug 14, 2025 •

edited

Loading

pytorch-bot bot commented Aug 14, 2025 •

edited

Loading