[Inductor] External callable registration API for Matmul tuning candidates #130774

maxyanghu · 2024-07-15T22:09:25Z

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @ColinPeppler @amjames @desertfire @chauhang

pytorch-bot · 2024-07-15T22:09:29Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/130774

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit f7e07bf with merge base b35f70d ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

linux-foundation-easycla · 2024-07-15T22:09:30Z

The committers listed above are authorized under a signed CLA.

✅ login: maxyanghu / name: Max Hu (41145f7, c83a47f, cca9969, 4985951, 8ccdbbd, 550c286, df8d1b6, eb25b8b)
✅ login: jansel / name: Jason Ansel (5a01ead, f7e07bf)

jansel

Please add a test.

torch/_inductor/external_callable.py

jansel · 2024-07-15T22:38:32Z

torch/_inductor/kernel/mm.py

+    from ..external_callable import external_matmul
+
+    for k in external_matmul:
+        choices.append(ExternKernelChoice(k).bind((mat1, mat2), layout))


We should only construct ExternKernelChoice once, since I believe the constructor here registers a new name. Maybe we need a functools.lru_cache(None) to dedupe these.

Hi @jansel , I see that ExternKernelChoice registers a new name for each new function. And we could run into duplication problem. But could you elaborate more how to use functools.lrucache(None) to dedupe these new name?

@funtools.lru_cache(None) def lazy_register_extern_choice(fn): return ExternKernelChoice(fn)

will cause it to only get constructed once.

Added deduplication class

maxyanghu · 2024-07-18T03:18:50Z

Hi Jason, I added a test and moved the list to config.py.

test/inductor/test_external_callables.py

torch/_inductor/kernel/mm.py

jansel · 2024-07-27T01:19:45Z

@pytorchbot merge

pytorchmergebot · 2024-07-27T01:21:42Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2024-07-27T02:18:27Z

Merge failed

Reason: 1 jobs have failed, first few of them are: trunk / linux-focal-cuda12.4-py3.10-gcc9-experimental-split-build-test / test (default, 1, 5, linux.4xlarge.nvidia.gpu)

Details for Dev Infra team

Raised by workflow job

maxyanghu · 2024-07-29T16:17:34Z

@jansel Could you help me with the failed test:
RuntimeError: Found Tesla M60 which is too old to be supported by the triton GPU compiler, which is used as the backend. Triton only supports devices of CUDA Capability >= 7.0, but your device is of CUDA capability 5.2
Seems that I need to skip tests on Tesla M60? Is there a way to do it?

Also test_flex_attention case is failing. I don't think it has anything to do with my PR.

Do I need to update my upstream?

jansel · 2024-07-30T02:54:21Z

Yes, you need to skip the test. I think the other inductor tests already skip M60 gpus, so you can look at test_torchinductor.py for an example.

github-actions · 2024-09-28T16:37:09Z

Looks like this PR hasn't been updated in a while so we're going to go ahead and mark this as Stale.
Feel free to remove the Stale label if you feel this was a mistake.
If you are unable to remove the Stale label please contact a maintainer in order to do so.
If you want the bot to never mark this PR stale again, add the no-stale label.
Stale pull requests will automatically be closed after 30 days of inactivity.

jansel · 2024-09-28T21:22:06Z

@pytorchbot rebase

pytorchmergebot · 2024-09-28T21:23:31Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2024-09-28T21:23:34Z

Successfully rebased external-registration onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout external-registration && git pull --rebase)

jansel · 2024-09-28T21:24:04Z

@pytorchbot merge

pytorchmergebot · 2024-09-28T21:25:50Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2024-09-28T21:31:05Z

Merge failed

Reason: 1 jobs have failed, first few of them are: Apply lint suggestions / lintrunner-autoformat

Details for Dev Infra team

Raised by workflow job

jansel · 2024-09-28T21:46:58Z

@pytorchbot merge

pytorchmergebot · 2024-09-28T21:48:36Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2024-09-28T22:09:21Z

Merge failed

Reason: 1 mandatory check(s) failed. The first few are:

Lint / lintrunner-noclang / linux-job

Dig deeper by viewing the failures on hud

Details for Dev Infra team

Raised by workflow job

Failing merge rule: Core Maintainers

jansel · 2024-10-02T04:46:05Z

@pytorchbot merge

pytorchmergebot · 2024-10-02T04:48:22Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2024-10-02T05:09:07Z

Merge failed

Reason: 1 jobs have failed, first few of them are: trunk / win-vs2019-cuda12.1-py3 / build

Details for Dev Infra team

Raised by workflow job

jansel · 2024-10-02T15:30:52Z

@pytorchbot merge

pytorchmergebot · 2024-10-02T15:32:50Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorch-bot bot added the module: inductor label Jul 15, 2024

maxyanghu mentioned this pull request Jul 15, 2024

[Inductor] Add an API to register external callable candidates for inductor's Matmul/Conv tuning choices #130769

Open

pytorchbot added the open source label Jul 15, 2024

jansel requested changes Jul 15, 2024

View reviewed changes

jansel reviewed Jul 19, 2024

View reviewed changes

test/inductor/test_external_callables.py Show resolved Hide resolved

maxyanghu marked this pull request as ready for review July 22, 2024 20:46

albanD added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Jul 23, 2024

jansel requested changes Jul 24, 2024

View reviewed changes

torch/_inductor/kernel/mm.py Outdated Show resolved Hide resolved

maxyanghu requested a review from jansel July 24, 2024 15:35

jansel approved these changes Jul 27, 2024

View reviewed changes

jansel added the release notes: inductor label Jul 27, 2024

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Jul 27, 2024

pytorchmergebot added the merging label Jul 27, 2024

pytorchmergebot removed the merging label Jul 27, 2024

github-actions bot added the Stale label Sep 28, 2024

jansel approved these changes Sep 28, 2024

View reviewed changes

maxyanghu added 3 commits September 28, 2024 21:23

Add external registration API for matmul candidates

41145f7

lint

4985951

add test and move external_matmul to config.py

c83a47f

maxyanghu added 2 commits September 28, 2024 21:23

skip CUDA compability < 7.0

8ccdbbd

lint

df8d1b6

pytorchmergebot force-pushed the external-registration branch from c765c6b to df8d1b6 Compare September 28, 2024 21:23

jansel removed the Stale label Sep 28, 2024

pytorchmergebot added the merging label Sep 28, 2024

pytorchmergebot removed the merging label Sep 28, 2024

lints

5a01ead

pytorchmergebot added the merging label Sep 28, 2024

pytorchmergebot removed the merging label Sep 28, 2024

lints

f7e07bf

pytorchmergebot added the merging label Oct 2, 2024

pytorchmergebot removed the merging label Oct 2, 2024

pytorchmergebot added the merging label Oct 2, 2024

pytorchmergebot added the Merged label Oct 2, 2024

pytorchmergebot closed this in a954a9e Oct 2, 2024

pytorchmergebot removed the merging label Oct 2, 2024

msaroufim mentioned this pull request Apr 23, 2025

RFC: The State of Custom CUDA extensions in PyTorch #152032

Open

[Inductor] External callable registration API for Matmul tuning candidates #130774

[Inductor] External callable registration API for Matmul tuning candidates #130774

Uh oh!

Conversation

maxyanghu commented Jul 15, 2024 • edited by pytorch-bot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Jul 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/130774

✅ No Failures

Uh oh!

linux-foundation-easycla bot commented Jul 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jansel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jansel Jul 15, 2024

Choose a reason for hiding this comment

Uh oh!

maxyanghu Jul 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jansel Jul 19, 2024

Choose a reason for hiding this comment

Uh oh!

maxyanghu Jul 22, 2024

Choose a reason for hiding this comment

Uh oh!

maxyanghu commented Jul 18, 2024

Uh oh!

Uh oh!

Uh oh!

jansel commented Jul 27, 2024

Uh oh!

pytorchmergebot commented Jul 27, 2024

Merge started

Uh oh!

pytorchmergebot commented Jul 27, 2024

Merge failed

Uh oh!

maxyanghu commented Jul 29, 2024

Uh oh!

jansel commented Jul 30, 2024

Uh oh!

github-actions bot commented Sep 28, 2024

Uh oh!

jansel commented Sep 28, 2024

Uh oh!

pytorchmergebot commented Sep 28, 2024

Uh oh!

pytorchmergebot commented Sep 28, 2024

Uh oh!

jansel commented Sep 28, 2024

Uh oh!

pytorchmergebot commented Sep 28, 2024

Merge started

Uh oh!

pytorchmergebot commented Sep 28, 2024

Merge failed

Uh oh!

jansel commented Sep 28, 2024

Uh oh!

pytorchmergebot commented Sep 28, 2024

Merge started

Uh oh!

pytorchmergebot commented Sep 28, 2024

Merge failed

Uh oh!

jansel commented Oct 2, 2024

Uh oh!

pytorchmergebot commented Oct 2, 2024

Merge started

Uh oh!

pytorchmergebot commented Oct 2, 2024

Merge failed

maxyanghu commented Jul 15, 2024 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Jul 15, 2024 •

edited

Loading

linux-foundation-easycla bot commented Jul 15, 2024 •

edited

Loading

maxyanghu Jul 18, 2024 •

edited

Loading