[sparse] Add padding for dense matrices in semi-structured sparse #110583

jcaip · 2023-10-05T04:23:45Z

Stack from ghstack (oldest at bottom):

Summary:

Currently we have shape constraints in semi-structured sparsity for both
CUTLASS and cuSPARSELt

These shape constraints unfortunately apply to both the dense and sparse
matrices in sparsedense matmul.

This PR adds in support for calling F.pad in order to pad dense
matrices to the right size with zeros and then pull out the
corresponding rows from the resultant result matrix.

We also throw a warning in this case.
The tests have also been updated to take in a dense_input_shape
parameter.

Test Plan:

python test/test_sparse_semi_structured.py

Reviewers:

Subscribers:

Tasks:

Tags:

Summary: Currently we have shape constraints in semi-structured sparsity for both CUTLASS and cuSPARSELt These shape constraints unfortunately apply to both the dense and sparse matrices in sparse@dense matmul. This PR adds in support for calling `F.pad` in order to pad dense matrices to the right size with zeros and then pull out the corresponding rows from the resultant result matrix. Test Plan: ``` python test/test_sparse_semi_structured.py -k pad ``` Reviewers: Subscribers: Tasks: Tags: [ghstack-poisoned]

pytorch-bot · 2023-10-05T04:23:48Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/110583

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 0d1277f with merge base 35750bf ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

Summary: Currently we have shape constraints in semi-structured sparsity for both CUTLASS and cuSPARSELt These shape constraints unfortunately apply to both the dense and sparse matrices in sparsedense matmul. This PR adds in support for calling `F.pad` in order to pad dense matrices to the right size with zeros and then pull out the corresponding rows from the resultant result matrix. Test Plan: ``` python test/test_sparse_semi_structured.py -k pad ``` Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: 34916c7 Pull Request resolved: #110583

… sparse" Summary: Currently we have shape constraints in semi-structured sparsity for both CUTLASS and cuSPARSELt These shape constraints unfortunately apply to both the dense and sparse matrices in sparsedense matmul. This PR adds in support for calling `F.pad` in order to pad dense matrices to the right size with zeros and then pull out the corresponding rows from the resultant result matrix. Test Plan: ``` python test/test_sparse_semi_structured.py -k pad ``` Reviewers: Subscribers: Tasks: Tags: [ghstack-poisoned]

… sparse" Summary: Currently we have shape constraints in semi-structured sparsity for both CUTLASS and cuSPARSELt These shape constraints unfortunately apply to both the dense and sparse matrices in sparsedense matmul. This PR adds in support for calling `F.pad` in order to pad dense matrices to the right size with zeros and then pull out the corresponding rows from the resultant result matrix. We also throw a warning in this case. The tests have also been updated to take in a dense_input_shape parameter. Test Plan: ``` python test/test_sparse_semi_structured.py ``` Reviewers: Subscribers: Tasks: Tags: [ghstack-poisoned]

Summary: Currently we have shape constraints in semi-structured sparsity for both CUTLASS and cuSPARSELt These shape constraints unfortunately apply to both the dense and sparse matrices in sparsedense matmul. This PR adds in support for calling `F.pad` in order to pad dense matrices to the right size with zeros and then pull out the corresponding rows from the resultant result matrix. We also throw a warning in this case. The tests have also been updated to take in a dense_input_shape parameter. Test Plan: ``` python test/test_sparse_semi_structured.py ``` Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: 609ede2 Pull Request resolved: #110583

cpuhrsch · 2023-10-09T16:49:39Z

torch/sparse/semi_structured.py

+        to_pad_m = -m % min_rows if m < min_rows or m % min_rows else 0
+        to_pad_n = -n % min_cols if n < min_cols or n % min_rows else 0
+        if to_pad_m or to_pad_n:
+            warnings.warn(


Will this warn on every invocation?

Yes, we can change this to on first invocation.

cc @cpuhrsch

Actually first invocation might be tricky because this is really a per-matmul warning. So we can't have a warning_shown class variable flag.

And since we create a new instance of the subclass we cannot store if the warning is shown in an instance variable. I will probably add a flag to disable all padding warnings instead. Perhaps we should make a padding_mode flag that needs to be set before we automatically do padding.

I'd say we remove the warning for now. If someone can't use the 2:4 format because their inputs don't align they have to pad. If it's slower because of padding, then they'll need to adjust inputs. But at least they can use it and try.

But a warning on every invocation can cause a lot of issues and slowdowns (I think even graph breaks maybe?)

cpuhrsch · 2023-10-09T16:50:40Z

torch/sparse/semi_structured.py

+                ),
+                UserWarning,
+            )
+            return torch.nn.functional.pad(original_tensor, (0, to_pad_n, 0, to_pad_m))


I think (hope) F.pad is smart enough to do nothing when to_pad is 0.

Also, can you try this code with torch.compile to see how the compiler handles the padding?

Yes, I will run our benchmarks again too to make sure there is no perf regression.

For torch.compile we don't have brians subclassing support yet.
Do you want me to just verify that we can fuse padding with a test?

I can also just open a new branch with the same padding changes in saf, if you want to try this e2e

@jcaip I think Brian's work actually just recently landed (at least I thought it did)

Oh perfect, I'll try that then

alexsamardzic

LGTM.

… sparse" Summary: Currently we have shape constraints in semi-structured sparsity for both CUTLASS and cuSPARSELt These shape constraints unfortunately apply to both the dense and sparse matrices in sparsedense matmul. This PR adds in support for calling `F.pad` in order to pad dense matrices to the right size with zeros and then pull out the corresponding rows from the resultant result matrix. We also throw a warning in this case. The tests have also been updated to take in a dense_input_shape parameter. Test Plan: ``` python test/test_sparse_semi_structured.py ``` Reviewers: Subscribers: Tasks: Tags: [ghstack-poisoned]

Summary: Currently we have shape constraints in semi-structured sparsity for both CUTLASS and cuSPARSELt These shape constraints unfortunately apply to both the dense and sparse matrices in sparsedense matmul. This PR adds in support for calling `F.pad` in order to pad dense matrices to the right size with zeros and then pull out the corresponding rows from the resultant result matrix. We also throw a warning in this case. The tests have also been updated to take in a dense_input_shape parameter. Test Plan: ``` python test/test_sparse_semi_structured.py ``` Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: 8251062 Pull Request resolved: #110583

… sparse" Summary: Currently we have shape constraints in semi-structured sparsity for both CUTLASS and cuSPARSELt These shape constraints unfortunately apply to both the dense and sparse matrices in sparsedense matmul. This PR adds in support for calling `F.pad` in order to pad dense matrices to the right size with zeros and then pull out the corresponding rows from the resultant result matrix. We also throw a warning in this case. The tests have also been updated to take in a dense_input_shape parameter. Test Plan: ``` python test/test_sparse_semi_structured.py ``` Reviewers: Subscribers: Tasks: Tags: [ghstack-poisoned]

Summary: Currently we have shape constraints in semi-structured sparsity for both CUTLASS and cuSPARSELt These shape constraints unfortunately apply to both the dense and sparse matrices in sparsedense matmul. This PR adds in support for calling `F.pad` in order to pad dense matrices to the right size with zeros and then pull out the corresponding rows from the resultant result matrix. We also throw a warning in this case. The tests have also been updated to take in a dense_input_shape parameter. Test Plan: ``` python test/test_sparse_semi_structured.py ``` Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: cf83333 Pull Request resolved: #110583

… sparse" Summary: Currently we have shape constraints in semi-structured sparsity for both CUTLASS and cuSPARSELt These shape constraints unfortunately apply to both the dense and sparse matrices in sparsedense matmul. This PR adds in support for calling `F.pad` in order to pad dense matrices to the right size with zeros and then pull out the corresponding rows from the resultant result matrix. We also throw a warning in this case. The tests have also been updated to take in a dense_input_shape parameter. Test Plan: ``` python test/test_sparse_semi_structured.py ``` Reviewers: Subscribers: Tasks: Tags: [ghstack-poisoned]

jcaip · 2023-10-10T20:55:07Z

I see no perf difference for padding no-op on nvidia-bert benchmarks

with padding:

  test_function     m     k      n          dtype  backend  sparse_latency (ms)  dense_latency (ms)  speedup (d/s)  correct  contiguous
0        linear  3072  1024  16384  torch.float16  cutlass             0.310758            0.447545       1.440169     True       False
1        linear  4096  1024  16384  torch.float16  cutlass             0.410349            0.586863       1.430155     True       False
2        linear  1024  1024  16384  torch.float16  cutlass             0.112575            0.178476       1.585390     True       False
3        linear  1024  4096  16384  torch.float16  cutlass             0.373492            0.560101       1.499633     True       False

no padding:

  test_function     m     k      n          dtype  backend  sparse_latency (ms)  dense_latency (ms)  speedup (d/s)  correct  contiguous
0        linear  3072  1024  16384  torch.float16  cutlass             0.311111            0.448242       1.440778     True       False
1        linear  4096  1024  16384  torch.float16  cutlass             0.411172            0.593480       1.443387     True       False
2        linear  1024  1024  16384  torch.float16  cutlass             0.113998            0.178642       1.567061     True       False
3        linear  1024  4096  16384  torch.float16  cutlass             0.375681            0.560720       1.492543     True       False

pytorchmergebot · 2023-10-13T07:34:04Z

Merge failed

Reason: New commits were pushed while merging. Please rerun the merge command.

Details for Dev Infra team

Raised by workflow job

jcaip · 2023-10-13T07:38:24Z

@pytorchbot merge

pytorchmergebot · 2023-10-13T07:40:13Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2023-10-13T07:40:37Z

Merge failed

Reason: 1 jobs have failed, first few of them are: trunk / macos-12-py3-arm64 / build

Details for Dev Infra team

Raised by workflow job

jcaip · 2023-10-13T14:20:28Z

@pytorchbot merge

pytorchmergebot · 2023-10-13T14:22:30Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2023-10-13T14:23:36Z

Merge failed

Reason: Command git -C /home/runner/work/pytorch/pytorch cherry-pick -x 72fb02d242d9fb9ac18d33bfe77fed943e58c6fd returned non-zero exit code 1

Auto-merging test/test_sparse_semi_structured.py
CONFLICT (content): Merge conflict in test/test_sparse_semi_structured.py
error: could not apply 72fb02d242d... [sparse] Add padding for dense matrices in semi-structured sparse
hint: After resolving the conflicts, mark them with
hint: "git add/rm <pathspec>", then run
hint: "git cherry-pick --continue".
hint: You can instead skip this commit with "git cherry-pick --skip".
hint: To abort and get back to the state before "git cherry-pick",
hint: run "git cherry-pick --abort".

Details for Dev Infra team

Raised by workflow job

… sparse" Summary: Currently we have shape constraints in semi-structured sparsity for both CUTLASS and cuSPARSELt These shape constraints unfortunately apply to both the dense and sparse matrices in sparsedense matmul. This PR adds in support for calling `F.pad` in order to pad dense matrices to the right size with zeros and then pull out the corresponding rows from the resultant result matrix. We also throw a warning in this case. The tests have also been updated to take in a dense_input_shape parameter. Test Plan: ``` python test/test_sparse_semi_structured.py ``` Reviewers: Subscribers: Tasks: Tags: [ghstack-poisoned]

Summary: Currently we have shape constraints in semi-structured sparsity for both CUTLASS and cuSPARSELt These shape constraints unfortunately apply to both the dense and sparse matrices in sparsedense matmul. This PR adds in support for calling `F.pad` in order to pad dense matrices to the right size with zeros and then pull out the corresponding rows from the resultant result matrix. We also throw a warning in this case. The tests have also been updated to take in a dense_input_shape parameter. Test Plan: ``` python test/test_sparse_semi_structured.py ``` Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: 8c0b9ee Pull Request resolved: #110583

… sparse" Summary: Currently we have shape constraints in semi-structured sparsity for both CUTLASS and cuSPARSELt These shape constraints unfortunately apply to both the dense and sparse matrices in sparsedense matmul. This PR adds in support for calling `F.pad` in order to pad dense matrices to the right size with zeros and then pull out the corresponding rows from the resultant result matrix. We also throw a warning in this case. The tests have also been updated to take in a dense_input_shape parameter. Test Plan: ``` python test/test_sparse_semi_structured.py ``` Reviewers: Subscribers: Tasks: Tags: [ghstack-poisoned]

jcaip · 2023-10-13T19:04:19Z

@pytorchbot merge

pytorchmergebot · 2023-10-13T19:06:12Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

@bdhirsh

Summary: This PR adds in torch.compile support for semi-structured sparsity, using the subclass tracing @bdhirsh added. Based on wether we are using cuSPARSELt or CUTLASS, we return a different representation of the inner tensors. Test Plan: ``` python test/test_sparse_semi_structured.py -k compile ``` Reviewers: Subscribers: Tasks: Tags: Pull Request resolved: #111049 Approved by: https://github.com/cpuhrsch ghstack dependencies: #110583

@bdhirsh

Summary: This PR adds in torch.compile support for semi-structured sparsity, using the subclass tracing @bdhirsh added. Based on wether we are using cuSPARSELt or CUTLASS, we return a different representation of the inner tensors. Test Plan: ``` python test/test_sparse_semi_structured.py -k compile ``` Reviewers: Subscribers: Tasks: Tags: Pull Request resolved: #111049 Approved by: https://github.com/cpuhrsch ghstack dependencies: #110583

pytorch-bot bot added the release notes: sparse release notes category label Oct 5, 2023

jcaip requested review from alexsamardzic and cpuhrsch October 6, 2023 18:51

jcaip mentioned this pull request Oct 6, 2023

Add padding support for dense matrices in (semi-structured sparse @dense) matmul #110744

Closed

cpuhrsch reviewed Oct 9, 2023

View reviewed changes

alexsamardzic approved these changes Oct 9, 2023

View reviewed changes

jcaip mentioned this pull request Oct 9, 2023

[sparse] Fix semi-structured sparse shape mismatch bug #110420

Closed

jcaip force-pushed the gh/jcaip/45/head branch from 0e5a3f3 to e58cc1c Compare October 9, 2023 19:53

cpuhrsch approved these changes Oct 11, 2023

View reviewed changes

pytorchmergebot removed the merging label Oct 13, 2023

pytorchmergebot added the merging label Oct 13, 2023

pytorchmergebot removed the merging label Oct 13, 2023

pytorchmergebot added the merging label Oct 13, 2023

pytorchmergebot removed the merging label Oct 13, 2023

pytorchmergebot added the merging label Oct 13, 2023

pytorchmergebot added Merged and removed merging labels Oct 13, 2023

pytorchmergebot closed this in 8db72a4 Oct 13, 2023

facebook-github-bot deleted the gh/jcaip/45/head branch October 17, 2023 14:24

jcaip mentioned this pull request Dec 12, 2023

[Tracker] torch.sparse semi-structured 2.3 beta release #115662

Closed

[sparse] Add padding for dense matrices in semi-structured sparse #110583

[sparse] Add padding for dense matrices in semi-structured sparse #110583

Uh oh!

Conversation

jcaip commented Oct 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Oct 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/110583

✅ No Failures

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexsamardzic left a comment

Choose a reason for hiding this comment

Uh oh!

jcaip commented Oct 10, 2023

Uh oh!

pytorchmergebot commented Oct 13, 2023

Merge failed

Uh oh!

jcaip commented Oct 13, 2023

Uh oh!

pytorchmergebot commented Oct 13, 2023

Merge started

Uh oh!

pytorchmergebot commented Oct 13, 2023

Merge failed

Uh oh!

jcaip commented Oct 13, 2023

Uh oh!

pytorchmergebot commented Oct 13, 2023

Merge started

Uh oh!

pytorchmergebot commented Oct 13, 2023

Merge failed

Uh oh!

jcaip commented Oct 13, 2023

Uh oh!

pytorchmergebot commented Oct 13, 2023

Merge started

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jcaip commented Oct 5, 2023 •

edited

Loading

pytorch-bot bot commented Oct 5, 2023 •

edited

Loading