[CD] Enable pypi dependencies both for XPU linux and Windows whls #141135

chuanqi129 · 2024-11-20T16:49:06Z

Enable xpu runtime pypi packages as dependencies of XPU CD wheels both for Linux and Windows.
Fixes #135867
Works for #139722 and #114850

pytorch-bot · 2024-11-20T16:49:10Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/141135

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 11 New Failures

As of commit 6621b54 with merge base ce572fe ():

NEW FAILURES - The following jobs have failed:

linux-binary-libtorch-pre-cxx11 / libtorch-rocm6_1-shared-with-deps-pre-cxx11-test (gh)
gcc ABI 0 not as expected.
linux-binary-libtorch-pre-cxx11 / libtorch-rocm6_2-shared-with-deps-pre-cxx11-test (gh)
gcc ABI 0 not as expected.
linux-binary-manywheel / manywheel-py3_10-rocm6_1-test (gh)
+ [[ '' != 1 ]]
linux-binary-manywheel / manywheel-py3_10-rocm6_2-test (gh)
gcc ABI 0 not as expected.
linux-binary-manywheel / manywheel-py3_11-rocm6_1-test (gh)
gcc ABI 0 not as expected.
linux-binary-manywheel / manywheel-py3_11-rocm6_2-test (gh)
##[error]Process completed with exit code 1.
linux-binary-manywheel / manywheel-py3_12-rocm6_1-test (gh)
gcc ABI 0 not as expected.
linux-binary-manywheel / manywheel-py3_12-rocm6_2-test (gh)
gcc ABI 0 not as expected.
linux-binary-manywheel / manywheel-py3_9-rocm6_1-test (gh)
gcc ABI 0 not as expected.
linux-binary-manywheel / manywheel-py3_9-rocm6_2-test (gh)
gcc ABI 0 not as expected.
macos-arm64-binary-libtorch-cxx11-abi / libtorch-cpu-shared-with-deps-cxx11-abi-build (gh)
cp: /Users/runner/work/pytorch/pytorch/pytorch/any_wheel/torch/lib/libomp.dylib: No such file or directory

This comment was automatically generated by Dr. CI and updates every 15 minutes.

atalman · 2024-11-21T17:34:03Z

@chuanqi129 this looks good. After the pypi depenencies are uploaded. We would need to rebase and retest this PR

atalman · 2024-11-21T19:58:42Z

.github/scripts/generate_binary_build_matrix.py

        "nvidia-nvtx-cu12==12.6.77; platform_system == 'Linux' and platform_machine == 'x86_64' | "
        "nvidia-nvjitlink-cu12==12.6.77; platform_system == 'Linux' and platform_machine == 'x86_64'"
    ),
+    "xpu": (


Hi @chuanqi129 please add these packages here: https://github.com/pytorch/test-infra/edit/main/s3_management/manage.py to PACKAGE_ALLOW_LIST

And perhaps here as well: https://github.com/pytorch/test-infra/blob/main/s3_management/update_dependencies.py#L11
So we can update them via script.

Thanks @atalman help to point it out, the PR has been submitted pytorch/test-infra#5965. Please help to review it

chuanqi129 · 2024-11-29T02:43:46Z

@pytorchbot rebase -b main

pytorchmergebot · 2024-11-29T02:45:19Z

@pytorchbot started a rebase job onto refs/remotes/origin/main. Check the current status here

pytorchmergebot · 2024-11-29T02:45:22Z

Successfully rebased xpu_cd_pip onto refs/remotes/origin/main, please pull locally before adding more changes (for example, via git checkout xpu_cd_pip && git pull --rebase)

chuanqi129 · 2024-11-29T13:09:05Z

Hi @atalman, I have retriggered the CI test and all PR change related tests are passed, could you please help to review it again?

atalman

lgtm

atalman · 2024-11-29T21:33:25Z

@pytorchmergebot merge -f "lgtm, failures are not related"

pytorchmergebot · 2024-11-29T21:34:50Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

…torch#141135) Enable xpu runtime pypi packages as dependencies of XPU CD wheels both for Linux and Windows. Fixes pytorch#135867 Works for pytorch#139722 and pytorch#114850 Pull Request resolved: pytorch#141135 Approved by: https://github.com/atalman

Due to XPU CD wheels enabled pypi dependencies by #141135, so the PYTORCH_EXTRA_INSTALL_REQUIREMENTS has value for XPU CD wheel build. Works for #139722 and #114850 Fixes #143838 Pull Request resolved: #143839 Approved by: https://github.com/huydhn

Due to XPU CD wheels enabled pypi dependencies by #141135, so the PYTORCH_EXTRA_INSTALL_REQUIREMENTS has value for XPU CD wheel build. Works for #139722 and #114850 Fixes #143838 Pull Request resolved: #143839 Approved by: https://github.com/huydhn (cherry picked from commit 438698b)

[CD] Remove redundant triton dependency for xpu wheels (#143839) Due to XPU CD wheels enabled pypi dependencies by #141135, so the PYTORCH_EXTRA_INSTALL_REQUIREMENTS has value for XPU CD wheel build. Works for #139722 and #114850 Fixes #143838 Pull Request resolved: #143839 Approved by: https://github.com/huydhn (cherry picked from commit 438698b) Co-authored-by: chuanqiw <chuanqi.wang@intel.com>

pytorch-bot bot added the topic: not user facing topic category label Nov 20, 2024

chuanqi129 mentioned this pull request Nov 20, 2024

[XPU] Upgrade the XPU support packages version to 2025.0 in CI/CD #139722

Closed

18 tasks

pytorchbot added the open source label Nov 20, 2024

chuanqi129 force-pushed the xpu_cd_pip branch from dd67c57 to a33df6e Compare November 20, 2024 17:52

chuanqi129 added ciflow/binaries Trigger all binary build and upload jobs on the PR ciflow/xpu Run XPU CI tasks labels Nov 20, 2024

chuanqi129 force-pushed the xpu_cd_pip branch 2 times, most recently from 34af335 to 2d2a488 Compare November 21, 2024 08:21

chuanqi129 marked this pull request as ready for review November 21, 2024 08:21

chuanqi129 requested a review from a team as a code owner November 21, 2024 08:21

chuanqi129 requested a review from atalman November 21, 2024 08:21

chuanqi129 force-pushed the xpu_cd_pip branch 2 times, most recently from 79e0319 to fd826ed Compare November 21, 2024 14:35

atalman reviewed Nov 21, 2024

View reviewed changes

chuanqi129 force-pushed the xpu_cd_pip branch 5 times, most recently from 1fe010e to ec55550 Compare November 25, 2024 16:54

bdhirsh added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Nov 26, 2024

chuanqi129 force-pushed the xpu_cd_pip branch 2 times, most recently from 6be8a84 to b070428 Compare November 28, 2024 14:24

chuanqi129 added the ciflow/trunk Trigger trunk jobs on your pull request label Nov 28, 2024

chuanqi129 force-pushed the xpu_cd_pip branch from b070428 to 1e35aff Compare November 28, 2024 18:34

pytorchmergebot force-pushed the xpu_cd_pip branch from 1e35aff to 038a1bb Compare November 29, 2024 02:45

chuanqi129 requested a review from atalman November 29, 2024 03:34

chuanqi129 added ciflow/xpu Run XPU CI tasks and removed ciflow/xpu Run XPU CI tasks labels Nov 29, 2024

chuanqi129 added 5 commits November 29, 2024 10:00

[CD] Enable pypi dependencies both for XPU linux and Windows whls

2e943cb

Enable xpu linux CD wheel with pypi dependencies

008f8a4

Fix xpu version in Windows smoke test

f451232

Update the dependency version

c6dabd3

Remove unnecessary dependencies

6621b54

chuanqi129 force-pushed the xpu_cd_pip branch from 038a1bb to 6621b54 Compare November 29, 2024 09:00

atalman approved these changes Nov 29, 2024

View reviewed changes

pytorchmergebot added the merging label Nov 29, 2024

pytorchmergebot added the Merged label Nov 29, 2024

pytorchmergebot closed this in a23ac6f Nov 29, 2024

pytorchmergebot removed the merging label Nov 29, 2024

chuanqi129 mentioned this pull request Dec 25, 2024

[CD] Remove redundant triton dependency for xpu wheels #143839

Closed

pytorchbot mentioned this pull request Dec 30, 2024

[CD] Remove redundant triton dependency for xpu wheels #143983

Merged

[CD] Enable pypi dependencies both for XPU linux and Windows whls #141135

[CD] Enable pypi dependencies both for XPU linux and Windows whls #141135

Uh oh!

Conversation

chuanqi129 commented Nov 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Nov 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/141135

❌ 11 New Failures

Uh oh!

atalman commented Nov 21, 2024

Uh oh!

atalman Nov 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

atalman Nov 21, 2024

Choose a reason for hiding this comment

Uh oh!

chuanqi129 Nov 22, 2024

Choose a reason for hiding this comment

Uh oh!

chuanqi129 commented Nov 29, 2024

Uh oh!

pytorchmergebot commented Nov 29, 2024

Uh oh!

pytorchmergebot commented Nov 29, 2024

Uh oh!

chuanqi129 commented Nov 29, 2024

Uh oh!

atalman left a comment

Choose a reason for hiding this comment

Uh oh!

atalman commented Nov 29, 2024

Uh oh!

pytorchmergebot commented Nov 29, 2024

Merge started

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

chuanqi129 commented Nov 20, 2024 •

edited

Loading

pytorch-bot bot commented Nov 20, 2024 •

edited

Loading

atalman Nov 21, 2024 •

edited

Loading