[Quant][CPU] Enable fp8 qconv #157076
[Quant][CPU] Enable fp8 qconv #157076
Conversation
pytorch-bot
bot
added
module: cpu
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/157076
Note: Links to docs will display an error until the docs builds have been completed. ✅ You can merge normally! (1 Unrelated Failure)As of commit 0cc5b80 with merge base 84b77ec ( FLAKY - The following job failed but was likely due to flakiness present on trunk:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
Xia-Weiwen
requested review from
CaoE,
Valentine233,
leslie-fang-intel,
mingfeima and
yanbing-j
and removed request for
CaoE,
Valentine233,
leslie-fang-intel,
mingfeima and
yanbing-j
Xia-Weiwen
requested review from
CaoE,
Valentine233,
leslie-fang-intel,
mingfeima and
yanbing-j
Xia-Weiwen
requested review from
digantdesai,
jerryzh168,
jianyuh,
kimishpatel and
salilsdesai
as code owners
|
have we discussed about long term whether these quantized ops should live in pytorch or torchao? |
Not yet. Do you have a plan to move all these kernels to Torchao? And does torchao have a plan to build cpp kernels for X86 CPU by default? Thanks. |
|
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
this is in discussion I think, we can revisit moving these kernels to torchao in the future then. but I think in the end it makes sense to move |
Moving makes sense to me, too. Please let us know when you figure out a plan. Thanks. |
5 participants
Summary
Enable fp8 qconv on CPU. It's part of the plan to enable fp8 static quantization on CPU. This PR only adds FP8 support of the existing int8 qconv op. It does not add a new op nor does it affect frontend or quantization flow. The schema of the qconv op is not changed either.
So, the FP8 qconv shares the same op as INT8 qconv and the difference is that src/wei dtype is fp8 instead of int8. The output dtype can be fp8/float32/bfloat16. The implementation uses the oneDNN library.
Note:
OneDNN does not support quantized fp8 convolution until v3.9 but the version used in PyTorch is v3.7.2. So, the op goes to the reference kernel for now. And we have also update the oneDNN path so that it's compatible with the fp8 dtype. Once oneDNN is upgraded to v3.9 or newer, minimum changes are needed to enable the oneDNN path. And we have ensured that the behavior of the reference kernel is the same as the new oneDNN's implementation.
Test plan
cc @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10 @jerryzh168