inductor dtype propagation fixes #141495

eellison · 2024-11-25T17:51:31Z

Stack from ghstack (oldest at bottom):

-> inductor dtype propagation fixes #141495

Add in upcast_compute_type on creation of new tensors (loads, constants)
Fixes index_expr - right now we are sort of inconsistent in dtype and dont always respect the dtype specified. would be nice to fix but not doing in this pr.
bug fix in view dtype where we were always upcasting back to fp32 when input was in bf16/fp16. we should only be doing that if the output is also in bf16/fp16.
for masked, avoid calling dtype propagation and just use output dtype.

Turns on the runtime dtype verification for opinfo tests. The separate test file is still useful because we can use it for testing turning off codegen_upcast_to_fp32.

Follow ups:

We could consider requiring less explicit upcast_compute_types calls and do it automatically. That would potentially make things easier but be less flexible in the future. Maybe I should have done it this pr.
Be more consistent on our index expr dtype printing.

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @ColinPeppler @amjames @desertfire @chauhang @aakhundov

[ghstack-poisoned]

pytorch-bot · 2024-11-25T17:51:35Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/141495

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (1 Unrelated Failure)

As of commit 6b4362f with merge base 43afaa4 ():

UNSTABLE - The following job failed but was likely due to flakiness present on trunk and has been marked as unstable:

inductor / cuda12.1-py3.10-gcc9-sm86 / test (inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu, unstable) (gh) (#141703)
convnext_base

This comment was automatically generated by Dr. CI and updates every 15 minutes.

ghstack-source-id: 533c286 Pull Request resolved: #141495

torch/_inductor/graph.py

cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy yf225 chenyang78 kadeng muchulee8 ColinPeppler amjames desertfire chauhang aakhundov [ghstack-poisoned]

ghstack-source-id: 844ca3f Pull Request resolved: #141495

cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy yf225 chenyang78 kadeng muchulee8 ColinPeppler amjames desertfire chauhang aakhundov [ghstack-poisoned]

- Add in upcast_compute_type on creation of new tensors (loads, constants) - Fixes index_expr - right now we are sort of inconsistent in dtype and dont always respect the dtype specified. would be nice to fix but not doing in this pr. - bug fix in view dtype where we were always upcasting back to fp32 when input was in bf16/fp16. we should only be doing that if the output is also in bf16/fp16. - for masked, avoid calling dtype propagation and just use output dtype. Turns on the runtime dtype verification for opinfo tests. The separate test file is still useful because we can use it for testing turning off codegen_upcast_to_fp32. cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy yf225 chenyang78 kadeng muchulee8 ColinPeppler amjames desertfire chauhang aakhundov [ghstack-poisoned]

blaine-rister

LGTM! Left a few tiny style nits.

torch/_inductor/dtype_propagation.py

torch/_inductor/utils.py

[ghstack-poisoned]

pytorchmergebot · 2024-11-27T23:43:31Z

Merge failed

Reason: New commits were pushed while merging. Please rerun the merge command.

Details for Dev Infra team

Raised by workflow job

eellison · 2024-11-27T23:43:34Z

@pytorchbot merge

pytorchmergebot · 2024-11-27T23:45:31Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2024-11-28T01:36:40Z

Merge failed

Reason: 1 jobs have failed, first few of them are: inductor-rocm / rocm6.2-py3.10-inductor / test (inductor, 1, 2, linux.rocm.gpu.2)

Details for Dev Infra team

Raised by workflow job

[ghstack-poisoned]

ghstack-source-id: c3a546b Pull Request resolved: #141495

eellison · 2024-11-28T07:25:08Z

@pytorchbot merge

pytorchmergebot · 2024-11-28T07:26:54Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

atalman · 2024-12-02T17:57:39Z

@pytorchmergebot revert -c ghfirst -m "Breaks internal builds"

pytorchmergebot · 2024-12-02T17:59:16Z

@pytorchbot successfully started a revert job. Check the current status here.
Questions? Feedback? Please reach out to the PyTorch DevX Team

pytorchmergebot · 2024-12-02T17:59:28Z

Reverting PR 141495 failed

Reason: Command git -C /home/runner/work/pytorch/pytorch revert --no-edit f83361b2745e0f3232a211f5d2e5a464e02f6c1f returned non-zero exit code 1

Auto-merging test/inductor/test_torchinductor_opinfo.py
Auto-merging torch/_inductor/codegen/triton.py
CONFLICT (content): Merge conflict in torch/_inductor/codegen/triton.py
Auto-merging torch/_inductor/graph.py
error: could not revert f83361b2745... inductor dtype propagation fixes (#141495)
hint: After resolving the conflicts, mark them with
hint: "git add/rm <pathspec>", then run
hint: "git revert --continue".
hint: You can instead skip this commit with "git revert --skip".
hint: To abort and get back to the state before "git revert",
hint: run "git revert --abort".
hint: Disable this message with "git config advice.mergeConflict false"

Details for Dev Infra team

Raised by workflow job

Should fix compile time regression, it was doing fairly expensive meta programming in init and being instantiated multiple times. Pull Request resolved: #141882 Approved by: https://github.com/ezyang ghstack dependencies: #139945, #140057, #141495

@voznesenskym

- Set the dtype of "acc" appropriately so that epilogue fusion will have args with dtype - Update dtype propagation to use `type_to_dtype` instead of instantiating tensor - Throw if we have a string arg where we should have a proper CSEVariable, unless we're doing the Modification Subgraph thing which is nyi. everything else is appropriately typed (cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @ColinPeppler @amjames @desertfire @chauhang @aakhundov @drisspg ). Pull Request resolved: #141991 Approved by: https://github.com/drisspg ghstack dependencies: #139945, #140057, #141495, #141882

- Add in upcast_compute_type on creation of new tensors (loads, constants) - Fixes index_expr - right now we are sort of inconsistent in dtype and dont always respect the dtype specified. would be nice to fix but not doing in this pr. - bug fix in view dtype where we were always upcasting back to fp32 when input was in bf16/fp16. we should only be doing that if the output is also in bf16/fp16. - for masked, avoid calling dtype propagation and just use output dtype. Turns on the runtime dtype verification for opinfo tests. The separate test file is still useful because we can use it for testing turning off codegen_upcast_to_fp32. Follow ups: - We could consider requiring less explicit upcast_compute_types calls and do it automatically. That would potentially make things easier but be less flexible in the future. Maybe I should have done it this pr. - Be more consistent on our index expr dtype printing. Pull Request resolved: pytorch#141495 Approved by: https://github.com/blaine-rister, https://github.com/arui-meta, https://github.com/ezyang ghstack dependencies: pytorch#139945, pytorch#140057

Should fix compile time regression, it was doing fairly expensive meta programming in init and being instantiated multiple times. Pull Request resolved: pytorch#141882 Approved by: https://github.com/ezyang ghstack dependencies: pytorch#139945, pytorch#140057, pytorch#141495

@voznesenskym

- Set the dtype of "acc" appropriately so that epilogue fusion will have args with dtype - Update dtype propagation to use `type_to_dtype` instead of instantiating tensor - Throw if we have a string arg where we should have a proper CSEVariable, unless we're doing the Modification Subgraph thing which is nyi. everything else is appropriately typed (cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @ColinPeppler @amjames @desertfire @chauhang @aakhundov @drisspg ). Pull Request resolved: pytorch#141991 Approved by: https://github.com/drisspg ghstack dependencies: pytorch#139945, pytorch#140057, pytorch#141495, pytorch#141882

@voznesenskym

- Set the dtype of "acc" appropriately so that epilogue fusion will have args with dtype - Update dtype propagation to use `type_to_dtype` instead of instantiating tensor - Throw if we have a string arg where we should have a proper CSEVariable, unless we're doing the Modification Subgraph thing which is nyi. everything else is appropriately typed (cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @ColinPeppler @amjames @desertfire @chauhang @aakhundov @drisspg ). Pull Request resolved: pytorch#141991 Approved by: https://github.com/drisspg ghstack dependencies: pytorch#139945, pytorch#140057, pytorch#141495, pytorch#141882

more fixes

d3697f7

[ghstack-poisoned]

eellison mentioned this pull request Nov 25, 2024

Refactor dtype propagation #139945

Closed

pytorch-bot bot added ciflow/inductor module: inductor labels Nov 25, 2024

eellison mentioned this pull request Nov 25, 2024

Add remaining method and tests for dtype propagation #140057

Closed

eellison added a commit that referenced this pull request Nov 25, 2024

more fixes

0572da7

ghstack-source-id: 533c286 Pull Request resolved: #141495

Skylion007 reviewed Nov 25, 2024

View reviewed changes

torch/_inductor/graph.py Outdated Show resolved Hide resolved

Update on "more fixes"

df92db2

cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy yf225 chenyang78 kadeng muchulee8 ColinPeppler amjames desertfire chauhang aakhundov [ghstack-poisoned]

Update on "more fixes"

4a66d8c

cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy yf225 chenyang78 kadeng muchulee8 ColinPeppler amjames desertfire chauhang aakhundov [ghstack-poisoned]

Update on "more fixes"

560c7e6

cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy yf225 chenyang78 kadeng muchulee8 ColinPeppler amjames desertfire chauhang aakhundov [ghstack-poisoned]

Update on "more fixes"

3981d9d

cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy yf225 chenyang78 kadeng muchulee8 ColinPeppler amjames desertfire chauhang aakhundov [ghstack-poisoned]

Update on "more fixes"

5cf3195

cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy yf225 chenyang78 kadeng muchulee8 ColinPeppler amjames desertfire chauhang aakhundov [ghstack-poisoned]

eellison added a commit that referenced this pull request Nov 26, 2024

more fixes

e493fbe

ghstack-source-id: 844ca3f Pull Request resolved: #141495

Update on "more fixes"

25e4eff

cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy yf225 chenyang78 kadeng muchulee8 ColinPeppler amjames desertfire chauhang aakhundov [ghstack-poisoned]

Update on "more fixes"

0198a08

cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy yf225 chenyang78 kadeng muchulee8 ColinPeppler amjames desertfire chauhang aakhundov [ghstack-poisoned]

Update on "more fixes"

81fecea

cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy yf225 chenyang78 kadeng muchulee8 ColinPeppler amjames desertfire chauhang aakhundov [ghstack-poisoned]

eellison requested a review from blaine-rister November 26, 2024 17:30

eellison added topic: not user facing topic category keep-going Don't stop on first failure, keep running tests until the end labels Nov 26, 2024

blaine-rister approved these changes Nov 26, 2024

View reviewed changes

torch/_inductor/dtype_propagation.py Outdated Show resolved Hide resolved

torch/_inductor/utils.py Outdated Show resolved Hide resolved

Update

80e1749

[ghstack-poisoned]

pytorchmergebot removed the merging label Nov 27, 2024

pytorchmergebot added the merging label Nov 27, 2024

pytorchmergebot removed the merging label Nov 28, 2024

Update

6b4362f

[ghstack-poisoned]

eellison added a commit that referenced this pull request Nov 28, 2024

more fixes

b21b04a

ghstack-source-id: c3a546b Pull Request resolved: #141495

pytorchmergebot added the merging label Nov 28, 2024

pytorchmergebot added the Merged label Nov 28, 2024

pytorchmergebot closed this in f83361b Nov 28, 2024

pytorchmergebot removed the merging label Nov 28, 2024

etaf mentioned this pull request Dec 2, 2024

[Break XPU] Triton kernel generated for XPU can not pass runtime_triton_dtype_assert. #141861

Closed

github-actions bot deleted the gh/eellison/731/head branch January 2, 2025 02:04

eellison mentioned this pull request Mar 25, 2025

[Inductor] track block shape of intermediary variables #149905

Closed

inductor dtype propagation fixes #141495

inductor dtype propagation fixes #141495

Uh oh!

Conversation

eellison commented Nov 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Nov 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/141495

✅ You can merge normally! (1 Unrelated Failure)

Uh oh!

Uh oh!

blaine-rister left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

pytorchmergebot commented Nov 27, 2024

Merge failed

Uh oh!

eellison commented Nov 27, 2024

Uh oh!

pytorchmergebot commented Nov 27, 2024

Merge started

Uh oh!

pytorchmergebot commented Nov 28, 2024

Merge failed

Uh oh!

eellison commented Nov 28, 2024

Uh oh!

pytorchmergebot commented Nov 28, 2024

Merge started

Uh oh!

atalman commented Dec 2, 2024

Uh oh!

pytorchmergebot commented Dec 2, 2024

Uh oh!

pytorchmergebot commented Dec 2, 2024

Reverting PR 141495 failed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

eellison commented Nov 25, 2024 •

edited

Loading

pytorch-bot bot commented Nov 25, 2024 •

edited

Loading