[Quant] [PT2] Enable batchnorm in _move_exported_model_to_eval #114547

leslie-fang-intel · 2023-11-26T10:28:46Z

Stack from ghstack (oldest at bottom):

-> [Quant] [PT2] Enable batchnorm in _move_exported_model_to_eval #114547

Summary
Add standalone batchnorm into _move_exported_model_to_eval to move it from training mode into eval mode

Test Plan

python -m pytest test_mkldnn_pattern_matcher.py -k test_qat_bn_conv2d
python -u -m pytest -s -v test_quantize_pt2e.py -k test_bn_move_exported_model_to_eval

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler

Differential Revision: D51853407

[ghstack-poisoned]

pytorch-bot · 2023-11-26T10:28:49Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/114547

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 1248a8f with merge base 7bbc19a ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

**Summary** Add standalone batchnorm into `_move_exported_model_to_eval` to move it from training mode into eval mode **Test Plan** ``` python -m pytest test_mkldnn_pattern_matcher.py -k test_qat_bn_conv2d ``` cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx peterbell10 ipiszy yf225 chenyang78 kadeng muchulee8 aakhundov ColinPeppler [ghstack-poisoned]

…eval" **Summary** Add standalone batchnorm into `_move_exported_model_to_eval` to move it from training mode into eval mode **Test Plan** ``` python -m pytest test_mkldnn_pattern_matcher.py -k test_qat_bn_conv2d ``` cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx peterbell10 ipiszy yf225 chenyang78 kadeng muchulee8 aakhundov ColinPeppler [ghstack-poisoned]

ghstack-source-id: e967970 Pull Request resolved: pytorch#114547

jerryzh168 · 2023-11-28T01:00:51Z

cc @andrewor14 please take a look

…eval" **Summary** Add standalone batchnorm into `_move_exported_model_to_eval` to move it from training mode into eval mode **Test Plan** ``` python -m pytest test_mkldnn_pattern_matcher.py -k test_qat_bn_conv2d ``` cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx peterbell10 ipiszy yf225 chenyang78 kadeng muchulee8 aakhundov ColinPeppler [ghstack-poisoned]

ghstack-source-id: 67d7b9e Pull Request resolved: #114547

andrewor14

Hi @leslie-fang-intel, thanks for working on this. Could you also add a check for batchnorm in the existing test?

torch/ao/quantization/pt2e/eval_utils.py

test/quantization/pt2e/test_quantize_pt2e_qat.py

andrewor14 · 2023-11-28T20:55:30Z

Actually, can you paste here what the graphs look like before and after the switch? I would expect to see aten._native_batch_norm_legit(... train=True) before, and aten._native_batch_norm_legit_no_training after the call to move_exported_model_to_eval. Is this what you observe in the tests?

leslie-fang-intel · 2023-11-29T01:27:27Z

Actually, can you paste here what the graphs look like before and after the switch? I would expect to see aten._native_batch_norm_legit(... train=True) before, and aten._native_batch_norm_legit_no_training after the call to move_exported_model_to_eval. Is this what you observe in the tests?

UT added as python -u -m pytest -s -v test_quantize_pt2e.py -k test_bn_move_exported_model_to_eval.
Yes, that's what we see and take this UT as example:

Before the graph looks like

def forward(self, x):
    arg0, = fx_pytree.tree_flatten_spec(([x], {}), self._in_spec)
    arg0_1 = arg0
    _tensor_constant0 = self._tensor_constant0
    add_ = torch.ops.aten.add_.Tensor(_tensor_constant0, 1);  _tensor_constant0 = None
    empty = torch.ops.aten.empty.memory_format([0], dtype = torch.uint8, layout = torch.strided, device = device(type='cpu'))
    _param_constant0 = self._param_constant0
    _param_constant1 = self._param_constant1
    _tensor_constant1 = self._tensor_constant1
    _tensor_constant2 = self._tensor_constant2
    _native_batch_norm_legit = torch.ops.aten._native_batch_norm_legit.default(arg0_1, _param_constant0, _param_constant1, _tensor_constant1, _tensor_constant2, True, 0.1, 1e-05);  arg0_1 = _param_constant0 = _param_constant1 = _tensor_constant1 = _tensor_constant2 = None
    getitem = _native_batch_norm_legit[0]
    getitem_1 = _native_batch_norm_legit[1]
    getitem_2 = _native_batch_norm_legit[2];  _native_batch_norm_legit = None
    _param_constant2 = self._param_constant2
    _param_constant3 = self._param_constant3
    conv2d = torch.ops.aten.conv2d.default(getitem, _param_constant2, _param_constant3);  getitem = _param_constant2 = _param_constant3 = None
    _tensor_constant3 = self._tensor_constant3
    add__1 = torch.ops.aten.add_.Tensor(_tensor_constant3, 1);  _tensor_constant3 = None
    empty_1 = torch.ops.aten.empty.memory_format([0], dtype = torch.uint8, layout = torch.strided, device = device(type='cpu'))
    _param_constant4 = self._param_constant4
    _param_constant5 = self._param_constant5
    _tensor_constant4 = self._tensor_constant4
    _tensor_constant5 = self._tensor_constant5
    _native_batch_norm_legit_1 = torch.ops.aten._native_batch_norm_legit.default(conv2d, _param_constant4, _param_constant5, _tensor_constant4, _tensor_constant5, True, 0.1, 1e-05);  conv2d = _param_constant4 = _param_constant5 = _tensor_constant4 = _tensor_constant5 = None
    getitem_3 = _native_batch_norm_legit_1[0]
    getitem_4 = _native_batch_norm_legit_1[1]
    getitem_5 = _native_batch_norm_legit_1[2];  _native_batch_norm_legit_1 = None
    return pytree.tree_unflatten([getitem_3], self._out_spec)

After the graph looks like

def forward(self, x):
    arg0, = fx_pytree.tree_flatten_spec(([x], {}), self._in_spec)
    arg0_1 = arg0
    _param_constant0 = self._param_constant0
    _param_constant1 = self._param_constant1
    _tensor_constant1 = self._tensor_constant1
    _tensor_constant2 = self._tensor_constant2
    _param_constant2 = self._param_constant2
    _param_constant3 = self._param_constant3
    _native_batch_norm_legit_no_training = torch.ops.aten._native_batch_norm_legit_no_training.default(arg0_1, _param_constant0, _param_constant1, _tensor_constant1, _tensor_constant2, 0.1, 1e-05);  arg0_1 = _param_constant0 = _param_constant1 = _tensor_constant1 = _tensor_constant2 = None
    getitem_6 = _native_batch_norm_legit_no_training[0];  _native_batch_norm_legit_no_training = None
    conv2d = torch.ops.aten.conv2d.default(getitem_6, _param_constant2, _param_constant3);  getitem_6 = _param_constant2 = _param_constant3 = None
    _param_constant4 = self._param_constant4
    _param_constant5 = self._param_constant5
    _tensor_constant4 = self._tensor_constant4
    _tensor_constant5 = self._tensor_constant5
    _native_batch_norm_legit_no_training_1 = torch.ops.aten._native_batch_norm_legit_no_training.default(conv2d, _param_constant4, _param_constant5, _tensor_constant4, _tensor_constant5, 0.1, 1e-05);  conv2d = _param_constant4 = _param_constant5 = _tensor_constant4 = _tensor_constant5 = None
    getitem_7 = _native_batch_norm_legit_no_training_1[0];  _native_batch_norm_legit_no_training_1 = None
    return pytree.tree_unflatten([getitem_7], self._out_spec)

…eval" **Summary** Add standalone batchnorm into `_move_exported_model_to_eval` to move it from training mode into eval mode **Test Plan** ``` python -m pytest test_mkldnn_pattern_matcher.py -k test_qat_bn_conv2d python -u -m pytest -s -v test_quantize_pt2e.py -k test_bn_move_exported_model_to_eval ``` cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx peterbell10 ipiszy yf225 chenyang78 kadeng muchulee8 aakhundov ColinPeppler [ghstack-poisoned]

#114547)" This reverts commit bab0540. Reverted #114547 on behalf of https://github.com/facebook-github-bot due to Diff reverted internally ([comment](#114547 (comment)))

…eval" **Summary** Add standalone batchnorm into `_move_exported_model_to_eval` to move it from training mode into eval mode **Test Plan** ``` python -m pytest test_mkldnn_pattern_matcher.py -k test_qat_bn_conv2d python -u -m pytest -s -v test_quantize_pt2e.py -k test_bn_move_exported_model_to_eval ``` cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx peterbell10 ipiszy yf225 chenyang78 kadeng muchulee8 aakhundov ColinPeppler [ghstack-poisoned]

ghstack-source-id: b75131f Pull Request resolved: #114547

leslie-fang-intel · 2023-12-02T01:46:09Z

Thanks @andrewor14, reopen this PR with additional changes in #114927. Could you help to check if it fixes the internal failures?

leslie-fang-intel · 2023-12-05T09:16:10Z

Hi @andrewor14, could you help to check if it fixes the internal failure?

andrewor14 · 2023-12-05T16:01:18Z

@andrewor14 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

andrewor14 · 2023-12-05T16:07:50Z

Hi @leslie-fang-intel, we're running the tests now. Please don't merge this one yet. I will merge it once all the internal tests pass / once we figure out all the test failures.

andrewor14 · 2023-12-06T19:48:59Z

@pytorchbot merge

pytorchmergebot · 2023-12-06T19:51:03Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

leslie-fang-intel · 2023-12-07T00:18:23Z

Hi @leslie-fang-intel, we're running the tests now. Please don't merge this one yet. I will merge it once all the internal tests pass / once we figure out all the test failures.

Thanks @andrewor14, BTW: we also plan to port it back to 2.2 branch, does it sound good to you?

…ch#114547) **Summary** Add standalone batchnorm into `_move_exported_model_to_eval` to move it from training mode into eval mode **Test Plan** ``` python -m pytest test_mkldnn_pattern_matcher.py -k test_qat_bn_conv2d python -u -m pytest -s -v test_quantize_pt2e.py -k test_bn_move_exported_model_to_eval ``` Pull Request resolved: pytorch#114547 Approved by: https://github.com/jgong5, https://github.com/andrewor14

…ytorch#114725) **Summary** Enable Inplace Dropout replacement in `_move_exported_model_to_eval` **Test Plan** ``` python -u -m pytest -s -v test_quantize_pt2e.py -k test_move_exported_model_to_eval ``` Pull Request resolved: pytorch#114725 Approved by: https://github.com/andrewor14, https://github.com/jgong5 ghstack dependencies: pytorch#114547

pytorch#114547)" This reverts commit bab0540. Reverted pytorch#114547 on behalf of https://github.com/facebook-github-bot due to Diff reverted internally ([comment](pytorch#114547 (comment)))

…ch#114547) **Summary** Add standalone batchnorm into `_move_exported_model_to_eval` to move it from training mode into eval mode **Test Plan** ``` python -m pytest test_mkldnn_pattern_matcher.py -k test_qat_bn_conv2d python -u -m pytest -s -v test_quantize_pt2e.py -k test_bn_move_exported_model_to_eval ``` Differential Revision: [D51853407](https://our.internmc.facebook.com/intern/diff/D51853407) Pull Request resolved: pytorch#114547 Approved by: https://github.com/jgong5, https://github.com/andrewor14

Enable batchnorm in _move_exported_model_to_eval

4d1513a

[ghstack-poisoned]

pytorch-bot bot added the release notes: AO frontend label Nov 26, 2023

leslie-fang-intel mentioned this pull request Nov 26, 2023

[Quant] [PT2] Fix an issue in Conv Binary Quantization Annotation #114540

Closed

leslie-fang-intel mentioned this pull request Nov 26, 2023

[Quant] [Inductor] Fix an issue in QConv Binary Pattern Match #114541

Closed

github-actions bot added release notes: quantization release notes category module: inductor labels Nov 26, 2023

leslie-fang-intel added ciflow/trunk Trigger trunk jobs on your pull request topic: not user facing topic category labels Nov 26, 2023

leslie-fang-intel requested review from andrewor14, jerryzh168 and jgong5 November 26, 2023 10:30

pytorchbot added the open source label Nov 26, 2023

jgong5 approved these changes Nov 27, 2023

View reviewed changes

This was referenced Nov 27, 2023

[Quant] Enable QConv2d with hardtanh post op #114578

Closed

[Quant] [PT2] Add Hardtanh and ReLU6 into X86InductorQuantizer Conv2d Unary Annotation #114579

Closed

[Quant] [Inductor] Enable the Inductor Lowering of QConv2d post op hardtanh #114580

Closed

leslie-fang-intel changed the title ~~Enable batchnorm in _move_exported_model_to_eval~~ [Quant] [PT2] Enable batchnorm in _move_exported_model_to_eval Nov 27, 2023

leslie-fang-intel added a commit to leslie-fang-intel/pytorch that referenced this pull request Nov 27, 2023

Enable batchnorm in _move_exported_model_to_eval

c9225a9

ghstack-source-id: e967970 Pull Request resolved: pytorch#114547

leslie-fang-intel added a commit that referenced this pull request Nov 28, 2023

Enable batchnorm in _move_exported_model_to_eval

9b6e280

ghstack-source-id: 67d7b9e Pull Request resolved: #114547

andrewor14 reviewed Nov 28, 2023

View reviewed changes

torch/ao/quantization/pt2e/eval_utils.py Show resolved Hide resolved

test/quantization/pt2e/test_quantize_pt2e_qat.py Show resolved Hide resolved

pytorchmergebot reopened this Dec 1, 2023

leslie-fang-intel added a commit that referenced this pull request Dec 2, 2023

Enable batchnorm in _move_exported_model_to_eval

999a899

ghstack-source-id: b75131f Pull Request resolved: #114547

pytorchmergebot added the merging label Dec 6, 2023

pytorchmergebot closed this in 1489e4b Dec 6, 2023

pytorchmergebot removed the merging label Dec 6, 2023

facebook-github-bot deleted the gh/leslie-fang-intel/48/head branch December 10, 2023 15:27

This was referenced Dec 13, 2023

[Cherry-pick] [Quant] [PT2] Enable batchnorm in _move_exported_model_to_eval #115715

Merged

[v.2.2.0] Release Tracker #115300

Closed

kit1980 removed the Reverted label Dec 21, 2023

[Quant] [PT2] Enable batchnorm in _move_exported_model_to_eval #114547

[Quant] [PT2] Enable batchnorm in _move_exported_model_to_eval #114547

Uh oh!

Conversation

leslie-fang-intel commented Nov 26, 2023 • edited by andrewor14 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Nov 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/114547

✅ No Failures

Uh oh!

jerryzh168 commented Nov 28, 2023

Uh oh!

andrewor14 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

andrewor14 commented Nov 28, 2023

Uh oh!

leslie-fang-intel commented Nov 29, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

leslie-fang-intel commented Dec 2, 2023

Uh oh!

leslie-fang-intel commented Dec 5, 2023

Uh oh!

andrewor14 commented Dec 5, 2023

Uh oh!

andrewor14 commented Dec 5, 2023

Uh oh!

andrewor14 commented Dec 6, 2023

Uh oh!

pytorchmergebot commented Dec 6, 2023

Merge started

Uh oh!

leslie-fang-intel commented Dec 7, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

leslie-fang-intel commented Nov 26, 2023 •

edited by andrewor14

Loading

pytorch-bot bot commented Nov 26, 2023 •

edited

Loading

leslie-fang-intel commented Nov 29, 2023 •

edited

Loading