Allow Int4WeightOnlyQuantizer to set different dtype for scales_and_zeros #479
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
scales_and_zeros As titled. Currently `Int4WeightOnlyQuantizer` is hardcoded to return `scales_and_zeros` with dtype `torch.bfloat16`. Adding `dtype` argument into the flow so that it can be different dtype.
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/479
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit f3c320a with merge base a35a1cd ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
facebook-github-bot
added
the
CLA Signed
larryliu0820
commented
Jul 5, 2024
| super().__init__() | ||
| self.padding = not _check_linear_int4_k(in_features, groupsize, inner_k_tiles) | ||
| if self.padding: | ||
| from model import find_multiple |
There was a problem hiding this comment.
I don't think there's a module called model
There was a problem hiding this comment.
Thanks I think this is a relic of when gptq was more deeply coupled with gpt-fast
|
This seems fine to merge although I do worry that most of our gptq tests are disabled right now in |
msaroufim
reviewed
Jul 5, 2024
msaroufim
approved these changes
Jul 5, 2024
I'm migrating torchchat to use these APIs, to be prepared for shared kernels across ET and PyTorch eager/compile. |
yanbing-j
pushed a commit
to yanbing-j/ao
that referenced
this pull request
Dec 9, 2024
readme update
yanbing-j
pushed a commit
to yanbing-j/ao
that referenced
this pull request
Dec 9, 2024
* Update quantize.py to use torchao Quantizers
Summary:
Remove duplicate code for Int4WeightOnlyQuantizer and
Int8DynActInt4WeightQuantizer and use torchao API.
Test Plan:
```
python torchchat.py generate llama2 --quantize '{"linear:int4": {"groupsize": 256}, "precision": {"dtype":"float16"}, "executor":{"accelerator":"cpu"}}' --prompt "Once upon a time," --max-new-tokens 256
python torchchat.py generate llama2 --quantize '{"linear:a8w4dq": {"groupsize": 256}, "precision": {"dtype":"float16"}, "executor":{"accelerator":"cpu"}}' --prompt "Once upon a time," --max-new-tokens 256
```
Reviewers:
Subscribers:
Tasks:
Tags:
* Fix import
Summary:
Test Plan:
Reviewers:
Subscribers:
Tasks:
Tags:
* Install torchao from gh
* Explain import
Summary:
Test Plan:
Reviewers:
Subscribers:
Tasks:
Tags:
* Fix dependencies
* Test ao PR pytorch#479
* Update torchao hash
* Update torchao pin
* Fix scheduler bf16/fp16 mix error
* Incorporate torchao changes
Summary:
Test Plan:
Reviewers:
Subscribers:
Tasks:
Tags:
* update hash
* Fix GPU CI job
Summary:
Test Plan:
Reviewers:
Subscribers:
Tasks:
Tags:
* More fix
* Fix executorch CI job
* Use quant api for int4 weight only quantization
Summary:
Test Plan:
Reviewers:
Subscribers:
Tasks:
Tags:
* Fix
Summary:
Test Plan:
Reviewers:
Subscribers:
Tasks:
Tags:
* Fix again
Summary:
Test Plan:
Reviewers:
Subscribers:
Tasks:
Tags:
* Fix 3
* Fix 4
* Try something
* debug
* Only migrate 8a4w
---------
Co-authored-by: Jack Zhang <jackzhxng@fb.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As titled. Currently
Int4WeightOnlyQuantizeris hardcoded to returnscales_and_zeroswith dtypetorch.bfloat16. Addingdtypeargument into the flow so that it can be different dtype.