Download pre-compiled AOTriton from GitHub unless AOTRITON_INSTALL_FROM_SOURCE=1 is set #136603

jithunnair-amd · 2024-09-25T03:46:52Z

PyTorch community members have reported issues with building PyTorch from source for ROCm in an environment that doesn't have aotriton pre-installed, because aotriton is only installed in the CI/manywheel docker images. Building aotriton from source can take ~45 minutes.

This PR fixes the issue by downloading the aotriton tarball in such scenarios, unless the user explicitly wants to build aotriton from source using the AOTRITON_INSTALL_FROM_SOURCE=1 env var

Two more issues this PR addresses: #136603 (comment) and #136603 (comment)

…URCE=1 is set

Otherwise bdist_wheel will miss libaotriton_v2.so

pytorch-bot · 2024-09-25T03:46:55Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/136603

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 5 New Failures, 6 Cancelled Jobs, 12 Unrelated Failures

As of commit e0c0948 with merge base e4cdc31 ():

NEW FAILURES - The following jobs have failed:

linux-aarch64-binary-manywheel / manywheel-py3_10-cuda-aarch64-build / build (gh)
ninja: build stopped: subcommand failed
linux-aarch64-binary-manywheel / manywheel-py3_11-cuda-aarch64-build / build (gh)
ninja: build stopped: subcommand failed
linux-aarch64-binary-manywheel / manywheel-py3_12-cuda-aarch64-build / build (gh)
ninja: build stopped: subcommand failed
linux-aarch64-binary-manywheel / manywheel-py3_9-cuda-aarch64-build / build (gh)
ninja: build stopped: subcommand failed
trunk / libtorch-linux-focal-cuda12.4-py3.7-gcc9-debug / build (gh)
/var/lib/jenkins/workspace/aten/src/ATen/native/sparse/cuda/SparseCsrTensorMath.cu:753:1094: error: ‘T* at::Tensor::data() const [with T = double]’ is deprecated: Tensor.data<T>() is deprecated. Please use Tensor.data_ptr<T>() instead. [-Werror=deprecated-declarations]

CANCELLED JOBS - The following jobs were cancelled. Please retry:

linux-s390x-binary-manywheel / manywheel-py3_10-cpu-s390x-build / build (gh)
##[error]The operation was canceled.
linux-s390x-binary-manywheel / manywheel-py3_11-cpu-s390x-build / build (gh)
##[error]The operation was canceled.
windows-binary-conda / conda-py3_11-cuda12_1-test (gh)
windows-binary-conda / conda-py3_11-cuda12_1-upload (gh)
windows-binary-conda / conda-py3_12-cuda12_1-test (gh)
windows-binary-conda / conda-py3_12-cuda12_1-upload (gh)

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

pull / linux-focal-py3.11-clang10 / test (dynamo, 3, 3, lf.linux.2xlarge) (gh) (disabled by #134602 but the issue was closed recently and a rebase is needed to make it pass)
test_transformers.py::TestSDPAPrivateUse1Only::test_scaled_dot_product_fused_attention_overrideable_backward
pull / linux-jammy-py3.10-clang15-asan / test (default, 2, 6, lf.linux.4xlarge) (gh) (disabled by #134600, #132861, #132862 but the issue was closed recently and a rebase is needed to make it pass)
test_transformers.py::TestSDPAPrivateUse1Only::test_fused_sdp_choice_privateuseone
windows-binary-conda / conda-py3_10-cuda11_8-test (gh) (detected as infra flaky with no log or failing log classifier)
windows-binary-conda / conda-py3_10-cuda12_1-test (gh) (detected as infra flaky with no log or failing log classifier)
windows-binary-conda / conda-py3_10-cuda12_4-test (gh) (detected as infra flaky with no log or failing log classifier)
windows-binary-conda / conda-py3_11-cuda11_8-test (gh) (detected as infra flaky with no log or failing log classifier)
windows-binary-conda / conda-py3_11-cuda12_4-test (gh) (detected as infra flaky with no log or failing log classifier)
windows-binary-conda / conda-py3_12-cuda11_8-test (gh) (detected as infra flaky with no log or failing log classifier)
windows-binary-conda / conda-py3_12-cuda12_4-test (gh) (detected as infra flaky with no log or failing log classifier)
windows-binary-conda / conda-py3_9-cuda11_8-test (gh) (detected as infra flaky with no log or failing log classifier)
windows-binary-conda / conda-py3_9-cuda12_1-test (gh) (detected as infra flaky with no log or failing log classifier)
windows-binary-conda / conda-py3_9-cuda12_4-test (gh) (detected as infra flaky with no log or failing log classifier)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

jithunnair-amd · 2024-09-26T05:47:45Z

ROCm CI regular as well as manylinux build jobs succeeded. Those test the preinstalled_aotriton scenario. I tested the build-from-source as well as the non-preinstalled, non-build-from-source scenarios locally and both installed the aotriton 0.7b version as expected.

jithunnair-amd · 2024-09-26T05:50:57Z

cmake/External/aotriton.cmake

+    install(DIRECTORY
+            $ENV{AOTRITON_INSTALLED_PREFIX}/lib
+            $ENV{AOTRITON_INSTALLED_PREFIX}/include
+            DESTINATION ${__AOTRITON_INSTALL_DIR})


This snippet adds the libaotriton_v2.so to the torch/lib directory so that it gets packaged with the pytorch wheel if building pytorch from source using a simple python setup.py bdist_wheel command, thus making the wheel portable.

jithunnair-amd · 2024-09-26T17:27:30Z

cmake/External/aotriton.cmake

+      INSTALL_DIR ${__AOTRITON_INSTALL_DIR}
      CMAKE_ARGS -DCMAKE_INSTALL_PREFIX:PATH=${__AOTRITON_INSTALL_DIR}
-      -DAOTRITON_COMPRESS_KERNEL=OFF
+      -DAOTRITON_COMPRESS_KERNEL=ON


Kernel compression results in a smaller aotriton shared library, avoiding linker errors when we have a large number of gfx targets being built for.

atalman

lgtm

jithunnair-amd · 2024-09-26T18:03:54Z

@pytorchbot merge -f "Unrelated CI failures"

pytorchmergebot · 2024-09-26T18:05:41Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

jithunnair-amd · 2024-09-26T19:39:58Z

@pytorchbot cherry-pick --onto release/2.5 -c critical

…OM_SOURCE=1 is set (#136603) PyTorch community members have reported issues with building PyTorch from source for ROCm in an environment that doesn't have aotriton pre-installed, because aotriton is only installed in the [CI](https://github.com/pytorch/pytorch/blob/a8ed873ba2aa13437a336863ae9d73c235798acc/.ci/docker/ubuntu-rocm/Dockerfile#L110)/[manywheel](https://github.com/pytorch/pytorch/blob/a8ed873ba2aa13437a336863ae9d73c235798acc/.ci/docker/manywheel/Dockerfile#L197) docker images. Building aotriton from source can take ~45 minutes. This PR fixes the issue by downloading the aotriton tarball in such scenarios, *unless the user explicitly wants to build aotriton from source using the AOTRITON_INSTALL_FROM_SOURCE=1 env var* Pull Request resolved: #136603 Approved by: https://github.com/atalman Co-authored-by: Xinya Zhang <Xinya.Zhang@amd.com> (cherry picked from commit 851b973)

pytorchbot · 2024-09-26T19:44:22Z

Cherry picking #136603

The cherry pick PR is at #136786 and it is recommended to link a critical cherry pick PR with an issue. The following tracker issues are updated:

[v.2.5.0] Release Tracker #135522 (comment)

Details for Dev Infra team

Raised by workflow job

malfet · 2024-09-26T20:26:22Z

@pytorchbot cherry-pick --onto release/2.5 -c critical

@jithunnair-amd can you please elaborate why this is a critical to cherry-pick into release branch?

jithunnair-amd · 2024-09-26T22:32:56Z

@pytorchbot cherry-pick --onto release/2.5 -c critical

@jithunnair-amd can you please elaborate why this is a critical to cherry-pick into release branch?

It fixes the issues PyTorch community users face when doing build-from-source of PyTorch on ROCm. I assumed that would come under "critical".

kit1980 · 2024-09-27T01:13:09Z

Is building from source relevant for release really?

jithunnair-amd · 2024-09-27T01:36:50Z

Is building from source relevant for release really?

I'll go with release management team's assessment on that. I assumed that we would want release branches to also build-from-source successfully, but if that's only considered important for main branch, we can close the other PR.

xinyazhang and others added 6 commits September 25, 2024 03:09

AOTriton: download binary from github unless AOTRITON_INSTALL_FROM_SO…

a833a89

…URCE=1 is set

Copy AOTRITON_INSTALLED_PREFIX/* to torch/

752f9a4

Otherwise bdist_wheel will miss libaotriton_v2.so

fixes

f5ba29a

Do not hardcode __AOTRITON_ARCH

fc17682

naming

18e9ffb

Revert Z changes to aotriton_version.txt

d650ce1

fixes

a781a4d

pytorchbot added the open source label Sep 25, 2024

jithunnair-amd added 2 commits September 25, 2024 04:42

fix rocm prefix

8be717e

Enable kernel compression for 0.7b to avoid linker errors

91e8af1

jithunnair-amd added release notes: rocm mandatorylabel ciflow/binaries Trigger all binary build and upload jobs on the PR ciflow/trunk Trigger trunk jobs on your pull request labels Sep 25, 2024

jithunnair-amd marked this pull request as ready for review September 25, 2024 20:11

jithunnair-amd requested review from atalman and malfet September 25, 2024 20:11

xinyazhang added 2 commits September 25, 2024 20:58

Set STAMP_DIR, otherwise cmake will try to use torch/src/aotriton-stamp

f9d5eec

Fix the use of PREFIX in ExternalProject_Add

e0c0948

jithunnair-amd commented Sep 26, 2024

View reviewed changes

atalman approved these changes Sep 26, 2024

View reviewed changes

pytorchmergebot added the merging label Sep 26, 2024

pytorchmergebot added the Merged label Sep 26, 2024

pytorchmergebot closed this in 851b973 Sep 26, 2024

pytorchmergebot removed the merging label Sep 26, 2024

pytorchbot mentioned this pull request Sep 26, 2024

[v.2.5.0] Release Tracker #135522

Closed

jithunnair-amd deleted the jnair/aotriton-cmake-update-only branch September 28, 2024 16:51

Download pre-compiled AOTriton from GitHub unless AOTRITON_INSTALL_FROM_SOURCE=1 is set #136603

Download pre-compiled AOTriton from GitHub unless AOTRITON_INSTALL_FROM_SOURCE=1 is set #136603

Uh oh!

Conversation

jithunnair-amd commented Sep 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Sep 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/136603

❌ 5 New Failures, 6 Cancelled Jobs, 12 Unrelated Failures

Uh oh!

jithunnair-amd commented Sep 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jithunnair-amd Sep 26, 2024

Choose a reason for hiding this comment

Uh oh!

jithunnair-amd Sep 26, 2024

Choose a reason for hiding this comment

Uh oh!

atalman left a comment

Choose a reason for hiding this comment

Uh oh!

jithunnair-amd commented Sep 26, 2024

Uh oh!

pytorchmergebot commented Sep 26, 2024

Merge started

Uh oh!

jithunnair-amd commented Sep 26, 2024

Uh oh!

pytorchbot commented Sep 26, 2024

Cherry picking #136603

Uh oh!

malfet commented Sep 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jithunnair-amd commented Sep 26, 2024

Uh oh!

kit1980 commented Sep 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jithunnair-amd commented Sep 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

jithunnair-amd commented Sep 25, 2024 •

edited

Loading

pytorch-bot bot commented Sep 25, 2024 •

edited

Loading

jithunnair-amd commented Sep 26, 2024 •

edited

Loading

malfet commented Sep 26, 2024 •

edited

Loading

kit1980 commented Sep 27, 2024 •

edited

Loading

jithunnair-amd commented Sep 27, 2024 •

edited

Loading