[CI] Add Compiled DDP / Compiled FSDP2 / compute-comm reordering tests to test_inductor_distributed #138178

yf225 · 2024-10-17T06:35:04Z

test_replicate_with_compiler.py and test_fully_shard_compile.py requires bf16, so needs to be run within test_inductor_distributed job (which uses A10G (SM80) and has bf16 support).

This allows us to migrate distributed jobs to T4 machines in #137161, as the compiled distributed jobs are the only blocking ones now.

Stack from ghstack (oldest at bottom):

[c10d][Partial-Graph Overlap] Support calling .wait_tensor() on output tensor of eager async_op=True collective if under allow_inflight_collective_as_graph_input_ctx() context manager #137763
[Distributed] Fix extra context on device 0 #135273
Upgrade distributed test to g4dn instances (T4 GPUs) #137161
-> [CI] Add Compiled DDP / Compiled FSDP2 / compute-comm reordering tests to test_inductor_distributed #138178

cc @XilunWu @H-Huang @awgu @kwen2501 @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k @c-p-i-o

…ibuted [ghstack-poisoned]

…ibuted ghstack-source-id: fdb8be4 Pull Request resolved: #138178

pytorch-bot · 2024-10-17T06:35:08Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/138178

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure

As of commit 7921287 with merge base 1f349ee ():

NEW FAILURE - The following job has failed:

periodic / win-vs2019-cuda11.8-py3 / test (force_on_cpu, 1, 1, windows.4xlarge.nonephemeral) (gh)
'Test'

This comment was automatically generated by Dr. CI and updates every 15 minutes.

yf225 · 2024-10-17T08:23:37Z

@pytorchbot merge

pytorchmergebot · 2024-10-17T08:25:37Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

kwen2501

LGTM

huydhn · 2024-10-17T17:29:59Z

@pytorchbot revert -m 'Sorry for reverting your change, but the new tests are failing inductor distributed jobs' -c nosignal

distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_backward_only GH job link HUD commit link

Let me add ciflow/inductor to the PR to get that signals

pytorchmergebot · 2024-10-17T17:31:54Z

@pytorchbot successfully started a revert job. Check the current status here.
Questions? Feedback? Please reach out to the PyTorch DevX Team

pytorchmergebot · 2024-10-17T17:32:09Z

@yf225 your PR has been successfully reverted.

…or_distributed (#138178)" This reverts commit 20af56d. Reverted #138178 on behalf of https://github.com/huydhn due to Sorry for reverting your change, but the new tests are failing inductor distributed jobs ([comment](#138178 (comment)))

…dering tests to test_inductor_distributed" `test_replicate_with_compiler.py` and `test_fully_shard_compile.py` requires bf16, so needs to be run within test_inductor_distributed job (which uses A10G (SM80) and has bf16 support). This allows us to migrate distributed jobs to T4 machines in #137161, as the compiled distributed jobs are the only blocking ones now. cc XilunWu H-Huang awgu kwen2501 wanchaol fegin fduwjj wz337 wconstab d4l3k c-p-i-o [ghstack-poisoned]

yf225 · 2024-10-20T04:48:20Z

@pytorchbot merge

pytorchmergebot · 2024-10-20T04:49:58Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2024-10-20T04:55:13Z

Merge failed

Reason: 1 jobs have failed, first few of them are: inductor-periodic / cuda12.1-py3.10-gcc9-sm80 / test (inductor_torchbench_smoketest_perf, 1, 1, linux.gcp.a100)

Details for Dev Infra team

Raised by workflow job

…dering tests to test_inductor_distributed" `test_replicate_with_compiler.py` and `test_fully_shard_compile.py` requires bf16, so needs to be run within test_inductor_distributed job (which uses A10G (SM80) and has bf16 support). This allows us to migrate distributed jobs to T4 machines in #137161, as the compiled distributed jobs are the only blocking ones now. cc XilunWu H-Huang awgu kwen2501 wanchaol fegin fduwjj wz337 wconstab d4l3k c-p-i-o [ghstack-poisoned]

yf225 · 2024-10-20T06:09:48Z

@pytorchbot merge

pytorchmergebot · 2024-10-20T06:11:34Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2024-10-20T07:14:12Z

Merge failed

Reason: 1 jobs have failed, first few of them are: inductor-periodic / cuda12.1-py3.10-gcc9-sm80 / test (inductor_torchbench_smoketest_perf, 1, 1, linux.gcp.a100)

Details for Dev Infra team

Raised by workflow job

yf225 · 2024-10-20T08:02:21Z

@pytorchbot merge

pytorchmergebot · 2024-10-20T08:04:00Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2024-10-20T09:53:33Z

Merge failed

Reason: 1 jobs have failed, first few of them are: periodic / linux-focal-rocm6.2-py3.10 / test (distributed, 1, 3, linux.rocm.gpu)

Details for Dev Infra team

Raised by workflow job

…dering tests to test_inductor_distributed" `test_replicate_with_compiler.py` and `test_fully_shard_compile.py` requires bf16, so needs to be run within test_inductor_distributed job (which uses A10G (SM80) and has bf16 support). This allows us to migrate distributed jobs to T4 machines in #137161, as the compiled distributed jobs are the only blocking ones now. cc XilunWu H-Huang awgu kwen2501 wanchaol fegin fduwjj wz337 wconstab d4l3k c-p-i-o [ghstack-poisoned]

yf225 · 2024-10-20T19:36:40Z

@pytorchbot merge -f "fixed the failing test, other tests are confirmed working by previous CI runs"

pytorchmergebot · 2024-10-20T19:38:10Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

[CI] Add Compiled DDP and Compiled FSDP2 tests to test_inductor_distr…

b5c6493

…ibuted [ghstack-poisoned]

yf225 requested a review from a team as a code owner October 17, 2024 06:35

yf225 added a commit that referenced this pull request Oct 17, 2024

[CI] Add Compiled DDP and Compiled FSDP2 tests to test_inductor_distr…

b293923

…ibuted ghstack-source-id: fdb8be4 Pull Request resolved: #138178

yf225 requested a review from xmfan October 17, 2024 06:37

yf225 added topic: not user facing topic category keep-going Don't stop on first failure, keep running tests until the end labels Oct 17, 2024

yf225 requested review from kwen2501 and seemethere October 17, 2024 06:43

xmfan approved these changes Oct 17, 2024

View reviewed changes

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Oct 17, 2024

pytorchmergebot added the merging label Oct 17, 2024

pytorchmergebot added the Merged label Oct 17, 2024

pytorchmergebot closed this in 20af56d Oct 17, 2024

pytorchmergebot removed the merging label Oct 17, 2024

kwen2501 reviewed Oct 17, 2024

View reviewed changes

huydhn removed the keep-going Don't stop on first failure, keep running tests until the end label Oct 17, 2024

pytorchmergebot added the Reverted label Oct 17, 2024

pytorchmergebot reopened this Oct 17, 2024

yf225 added ciflow/inductor keep-going Don't stop on first failure, keep running tests until the end and removed Reverted Merged labels Oct 17, 2024

yf225 removed Merged Reverted labels Oct 20, 2024

pytorchmergebot added the merging label Oct 20, 2024

pytorchmergebot removed the merging label Oct 20, 2024

pytorchmergebot added the merging label Oct 20, 2024

pytorchmergebot removed the merging label Oct 20, 2024

pytorchmergebot added the merging label Oct 20, 2024

pytorchmergebot removed the merging label Oct 20, 2024

yf225 added 2 commits October 20, 2024 02:58

pytorchmergebot added the merging label Oct 20, 2024

pytorchmergebot added the Merged label Oct 20, 2024

pytorchmergebot closed this in a9f4f89 Oct 20, 2024

pytorchmergebot removed the merging label Oct 20, 2024

github-actions bot deleted the gh/yf225/145/head branch November 20, 2024 02:07

[CI] Add Compiled DDP / Compiled FSDP2 / compute-comm reordering tests to test_inductor_distributed #138178

[CI] Add Compiled DDP / Compiled FSDP2 / compute-comm reordering tests to test_inductor_distributed #138178

Uh oh!

Conversation

yf225 commented Oct 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Oct 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/138178

❌ 1 New Failure

Uh oh!

yf225 commented Oct 17, 2024

Uh oh!

pytorchmergebot commented Oct 17, 2024

Merge started

Uh oh!

kwen2501 left a comment

Choose a reason for hiding this comment

Uh oh!

huydhn commented Oct 17, 2024

Uh oh!

pytorchmergebot commented Oct 17, 2024

Uh oh!

pytorchmergebot commented Oct 17, 2024

Uh oh!

yf225 commented Oct 20, 2024

Uh oh!

pytorchmergebot commented Oct 20, 2024

Merge started

Uh oh!

pytorchmergebot commented Oct 20, 2024

Merge failed

Uh oh!

yf225 commented Oct 20, 2024

Uh oh!

pytorchmergebot commented Oct 20, 2024

Merge started

Uh oh!

pytorchmergebot commented Oct 20, 2024

Merge failed

Uh oh!

yf225 commented Oct 20, 2024

Uh oh!

pytorchmergebot commented Oct 20, 2024

Merge started

Uh oh!

pytorchmergebot commented Oct 20, 2024

Merge failed

Uh oh!

yf225 commented Oct 20, 2024

Uh oh!

pytorchmergebot commented Oct 20, 2024

Merge started

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

yf225 commented Oct 17, 2024 •

edited

Loading

pytorch-bot bot commented Oct 17, 2024 •

edited

Loading