Update ONNX's IO Adapter to support FakeTensor with ExportedProgram #114407
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Currently, the ONNX exporter using torch.nn.Module as input can support FakeTensor because the ONNX model stores all initializers When using torch.export.ExportedProgram as input, the initializers are lifted as inputs. In order to execute the ONNX model, we need to pass a reference to the non-fake model to the ONNXProgram.adapt_torch_inputs_to_onnx API, so that initializers can be fetched from the model and fed to the ONNX model as input [ghstack-poisoned]
thiagocrepaldi
requested review from
BowenBao,
abock and
wschin
as code owners
pytorch-bot
bot
added
the
release notes: onnx
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/114407
Note: Links to docs will display an error until the docs builds have been completed. ⏳ No Failures, 4 PendingAs of commit 73d1f8b with merge base 441ecf0 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
…edProgram" Currently, the ONNX exporter using torch.nn.Module as input can support FakeTensor because the ONNX model stores all initializers When using torch.export.ExportedProgram as input, the initializers are lifted as inputs. In order to execute the ONNX model, we need to pass a reference to the non-fake model to the ONNXProgram.adapt_torch_inputs_to_onnx API, so that initializers can be fetched from the model and fed to the ONNX model as input [ghstack-poisoned]
thiagocrepaldi
pushed a commit
that referenced
this pull request
Nov 22, 2023
Currently, the ONNX exporter using torch.nn.Module as input can support FakeTensor because the ONNX model stores all initializers When using torch.export.ExportedProgram as input, the initializers are lifted as inputs. In order to execute the ONNX model, we need to pass a reference to the non-fake model to the ONNXProgram.adapt_torch_inputs_to_onnx API, so that initializers can be fetched from the model and fed to the ONNX model as input ghstack-source-id: 954c65e Pull Request resolved: #114407
BowenBao
reviewed
Nov 22, 2023
…edProgram" Currently, the ONNX exporter using torch.nn.Module as input can support FakeTensor because the ONNX model stores all initializers When using torch.export.ExportedProgram as input, the initializers are lifted as inputs. In order to execute the ONNX model, we need to pass a reference to the non-fake model to the ONNXProgram.adapt_torch_inputs_to_onnx API, so that initializers can be fetched from the model and fed to the ONNX model as input [ghstack-poisoned]
thiagocrepaldi
pushed a commit
that referenced
this pull request
Nov 28, 2023
Currently, the ONNX exporter using torch.nn.Module as input can support FakeTensor because the ONNX model stores all initializers When using torch.export.ExportedProgram as input, the initializers are lifted as inputs. In order to execute the ONNX model, we need to pass a reference to the non-fake model to the ONNXProgram.adapt_torch_inputs_to_onnx API, so that initializers can be fetched from the model and fed to the ONNX model as input ghstack-source-id: fdfe139 Pull Request resolved: #114407
…edProgram" Currently, the ONNX exporter using torch.nn.Module as input can support FakeTensor because the ONNX model stores all initializers When using torch.export.ExportedProgram as input, the initializers are lifted as inputs. In order to execute the ONNX model, we need to pass a reference to the non-fake model to the ONNXProgram.adapt_torch_inputs_to_onnx API, so that initializers can be fetched from the model and fed to the ONNX model as input [ghstack-poisoned]
thiagocrepaldi
pushed a commit
that referenced
this pull request
Nov 29, 2023
Currently, the ONNX exporter using torch.nn.Module as input can support FakeTensor because the ONNX model stores all initializers When using torch.export.ExportedProgram as input, the initializers are lifted as inputs. In order to execute the ONNX model, we need to pass a reference to the non-fake model to the ONNXProgram.adapt_torch_inputs_to_onnx API, so that initializers can be fetched from the model and fed to the ONNX model as input ghstack-source-id: 8011f60 Pull Request resolved: #114407
added 7 commits
December 1, 2023 20:37
…edProgram" Currently, the ONNX exporter using torch.nn.Module as input can support FakeTensor because the ONNX model stores all initializers When using torch.export.ExportedProgram as input, the initializers are lifted as inputs. In order to execute the ONNX model, we need to pass a reference to the non-fake model to the ONNXProgram.adapt_torch_inputs_to_onnx API, so that initializers can be fetched from the model and fed to the ONNX model as input [ghstack-poisoned]
…edProgram" Currently, the ONNX exporter using torch.nn.Module as input can support FakeTensor because the ONNX model stores all initializers When using torch.export.ExportedProgram as input, the initializers are lifted as inputs. In order to execute the ONNX model, we need to pass a reference to the non-fake model to the ONNXProgram.adapt_torch_inputs_to_onnx API, so that initializers can be fetched from the model and fed to the ONNX model as input [ghstack-poisoned]
…edProgram" Currently, the ONNX exporter using torch.nn.Module as input can support FakeTensor because the ONNX model stores all initializers When using torch.export.ExportedProgram as input, the initializers are lifted as inputs. In order to execute the ONNX model, we need to pass a reference to the non-fake model to the ONNXProgram.adapt_torch_inputs_to_onnx API, so that initializers can be fetched from the model and fed to the ONNX model as input [ghstack-poisoned]
…edProgram" Currently, the ONNX exporter using torch.nn.Module as input can support FakeTensor because the ONNX model stores all initializers When using torch.export.ExportedProgram as input, the initializers are lifted as inputs. In order to execute the ONNX model, we need to pass a reference to the non-fake model to the ONNXProgram.adapt_torch_inputs_to_onnx API, so that initializers can be fetched from the model and fed to the ONNX model as input [ghstack-poisoned]
…edProgram" Currently, the ONNX exporter using torch.nn.Module as input can support FakeTensor because the ONNX model stores all initializers When using torch.export.ExportedProgram as input, the initializers are lifted as inputs. In order to execute the ONNX model, we need to pass a reference to the non-fake model to the ONNXProgram.adapt_torch_inputs_to_onnx API, so that initializers can be fetched from the model and fed to the ONNX model as input [ghstack-poisoned]
…edProgram" Currently, the ONNX exporter using torch.nn.Module as input can support FakeTensor because the ONNX model stores all initializers When using torch.export.ExportedProgram as input, the initializers are lifted as inputs. In order to execute the ONNX model, we need to pass a reference to the non-fake model to the ONNXProgram.adapt_torch_inputs_to_onnx API, so that initializers can be fetched from the model and fed to the ONNX model as input [ghstack-poisoned]
…edProgram" Currently, the ONNX exporter using torch.nn.Module as input can support FakeTensor because the ONNX model stores all initializers When using torch.export.ExportedProgram as input, the initializers are lifted as inputs. In order to execute the ONNX model, we need to pass a reference to the non-fake model to the ONNXProgram.adapt_torch_inputs_to_onnx API, so that initializers can be fetched from the model and fed to the ONNX model as input [ghstack-poisoned]
thiagocrepaldi
commented
Dec 5, 2023
added 3 commits
December 5, 2023 22:33
…edProgram" Currently, the ONNX exporter using torch.nn.Module as input can support FakeTensor because the ONNX model stores all initializers When using torch.export.ExportedProgram as input, the initializers are lifted as inputs. In order to execute the ONNX model, we need to pass a reference to the non-fake model to the ONNXProgram.adapt_torch_inputs_to_onnx API, so that initializers can be fetched from the model and fed to the ONNX model as input [ghstack-poisoned]
…edProgram" Currently, the ONNX exporter using torch.nn.Module as input can support FakeTensor because the ONNX model stores all initializers When using torch.export.ExportedProgram as input, the initializers are lifted as inputs. In order to execute the ONNX model, we need to pass a reference to the non-fake model to the ONNXProgram.adapt_torch_inputs_to_onnx API, so that initializers can be fetched from the model and fed to the ONNX model as input [ghstack-poisoned]
…edProgram" Currently, the ONNX exporter using torch.nn.Module as input can support FakeTensor because the ONNX model stores all initializers When using torch.export.ExportedProgram as input, the initializers are lifted as inputs. In order to execute the ONNX model, we need to pass a reference to the non-fake model to the ONNXProgram.adapt_torch_inputs_to_onnx API, so that initializers can be fetched from the model and fed to the ONNX model as input [ghstack-poisoned]
pytorchmergebot
pushed a commit
that referenced
this pull request
Dec 9, 2023
#115281) Currently (after #114407), the user has must pass the original user ``model`` to APIs such as ``ONNXProgram.__call__``, ``ONNXProgram.adapt_torch_inputs_to_onnx`` and ``ONNXProgram.adapt_torch_outputs_to_onnx`` APIs. This was needed because when the model is fakefied, a version of the non-fakefied model is needed so that the Initializers, buffers and constants can be extracted from a real model (and used as input to the ONNX model). That approach brings an unnecessary usability burden to the user when the model is not fakefied, because the model that was already passed to ``torch.onnx.dynamo_export`` could be used to extract ``state_dict``. This PR adds ``ONNXProgram._model_torch`` attribute to store the user model and demote ``model`` argument of the aforementioned APIs to optional, only (as opposed to required). As a result, for the fakefied model scenario, the user still need to pass the required model, but for non fakefied models, the persisted model is implicitly used to extract the model state_dict, making it easier to use. Pull Request resolved: #115281 Approved by: https://github.com/BowenBao ghstack dependencies: #114407
thiagocrepaldi
pushed a commit
to thiagocrepaldi/pytorch
that referenced
this pull request
Dec 11, 2023
…ytorch#114407) Currently, the ONNX exporter using torch.nn.Module as input can support FakeTensor because the ONNX model stores all initializers When using torch.export.ExportedProgram as input, the initializers are lifted as inputs. In order to execute the ONNX model, we need to pass a reference to the non-fake model to the ONNXProgram.adapt_torch_inputs_to_onnx API, so that initializers can be fetched from the model and fed to the ONNX model as input ps: pytorch#115461 will track the API revision for the cases where additional `model_with_state_dict` are required to produce complete ONNX files exported with fake support. This is also tracked by the umbrella fake tensor issue pytorch#105464 FYI @BowenBao Pull Request resolved: pytorch#114407 Approved by: https://github.com/BowenBao
This was referenced Dec 11, 2023
Merged
dmenig
pushed a commit
to dmenig/pytorch
that referenced
this pull request
Dec 21, 2023
…ytorch#114407) Currently, the ONNX exporter using torch.nn.Module as input can support FakeTensor because the ONNX model stores all initializers When using torch.export.ExportedProgram as input, the initializers are lifted as inputs. In order to execute the ONNX model, we need to pass a reference to the non-fake model to the ONNXProgram.adapt_torch_inputs_to_onnx API, so that initializers can be fetched from the model and fed to the ONNX model as input ps: pytorch#115461 will track the API revision for the cases where additional `model_with_state_dict` are required to produce complete ONNX files exported with fake support. This is also tracked by the umbrella fake tensor issue pytorch#105464 FYI @BowenBao Pull Request resolved: pytorch#114407 Approved by: https://github.com/BowenBao
dmenig
pushed a commit
to dmenig/pytorch
that referenced
this pull request
Dec 21, 2023
pytorch#115281) Currently (after pytorch#114407), the user has must pass the original user ``model`` to APIs such as ``ONNXProgram.__call__``, ``ONNXProgram.adapt_torch_inputs_to_onnx`` and ``ONNXProgram.adapt_torch_outputs_to_onnx`` APIs. This was needed because when the model is fakefied, a version of the non-fakefied model is needed so that the Initializers, buffers and constants can be extracted from a real model (and used as input to the ONNX model). That approach brings an unnecessary usability burden to the user when the model is not fakefied, because the model that was already passed to ``torch.onnx.dynamo_export`` could be used to extract ``state_dict``. This PR adds ``ONNXProgram._model_torch`` attribute to store the user model and demote ``model`` argument of the aforementioned APIs to optional, only (as opposed to required). As a result, for the fakefied model scenario, the user still need to pass the required model, but for non fakefied models, the persisted model is implicitly used to extract the model state_dict, making it easier to use. Pull Request resolved: pytorch#115281 Approved by: https://github.com/BowenBao ghstack dependencies: pytorch#114407
dmenig
pushed a commit
to dmenig/pytorch
that referenced
this pull request
Dec 21, 2023
…orch#114762) Fixed by pytorch#113982 Pull Request resolved: pytorch#114762 Approved by: https://github.com/BowenBao ghstack dependencies: pytorch#114407, pytorch#115281
dmenig
pushed a commit
to dmenig/pytorch
that referenced
this pull request
Dec 21, 2023
…ytorch#115353) Pull Request resolved: pytorch#115353 Approved by: https://github.com/BowenBao ghstack dependencies: pytorch#114407, pytorch#115281, pytorch#114762
huydhn
pushed a commit
that referenced
this pull request
Jan 5, 2024
…114407) (#115578) Currently, the ONNX exporter using torch.nn.Module as input can support FakeTensor because the ONNX model stores all initializers When using torch.export.ExportedProgram as input, the initializers are lifted as inputs. In order to execute the ONNX model, we need to pass a reference to the non-fake model to the ONNXProgram.adapt_torch_inputs_to_onnx API, so that initializers can be fetched from the model and fed to the ONNX model as input ps: #115461 will track the API revision for the cases where additional `model_with_state_dict` are required to produce complete ONNX files exported with fake support. This is also tracked by the umbrella fake tensor issue #105464 FYI @BowenBao Pull Request resolved: #114407 Approved by: https://github.com/BowenBao
thiagocrepaldi
pushed a commit
to thiagocrepaldi/pytorch
that referenced
this pull request
Jan 6, 2024
pytorch#115281) Currently (after pytorch#114407), the user has must pass the original user ``model`` to APIs such as ``ONNXProgram.__call__``, ``ONNXProgram.adapt_torch_inputs_to_onnx`` and ``ONNXProgram.adapt_torch_outputs_to_onnx`` APIs. This was needed because when the model is fakefied, a version of the non-fakefied model is needed so that the Initializers, buffers and constants can be extracted from a real model (and used as input to the ONNX model). That approach brings an unnecessary usability burden to the user when the model is not fakefied, because the model that was already passed to ``torch.onnx.dynamo_export`` could be used to extract ``state_dict``. This PR adds ``ONNXProgram._model_torch`` attribute to store the user model and demote ``model`` argument of the aforementioned APIs to optional, only (as opposed to required). As a result, for the fakefied model scenario, the user still need to pass the required model, but for non fakefied models, the persisted model is implicitly used to extract the model state_dict, making it easier to use. Pull Request resolved: pytorch#115281 Approved by: https://github.com/BowenBao ghstack dependencies: pytorch#114407
huydhn
pushed a commit
that referenced
this pull request
Jan 8, 2024
#115281) (#115583) Currently (after #114407), the user has must pass the original user ``model`` to APIs such as ``ONNXProgram.__call__``, ``ONNXProgram.adapt_torch_inputs_to_onnx`` and ``ONNXProgram.adapt_torch_outputs_to_onnx`` APIs. This was needed because when the model is fakefied, a version of the non-fakefied model is needed so that the Initializers, buffers and constants can be extracted from a real model (and used as input to the ONNX model). That approach brings an unnecessary usability burden to the user when the model is not fakefied, because the model that was already passed to ``torch.onnx.dynamo_export`` could be used to extract ``state_dict``. This PR adds ``ONNXProgram._model_torch`` attribute to store the user model and demote ``model`` argument of the aforementioned APIs to optional, only (as opposed to required). As a result, for the fakefied model scenario, the user still need to pass the required model, but for non fakefied models, the persisted model is implicitly used to extract the model state_dict, making it easier to use. Pull Request resolved: #115281 Approved by: https://github.com/BowenBao ghstack dependencies: #114407
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Stack from ghstack (oldest at bottom):
Currently, the ONNX exporter using torch.nn.Module as input can support
FakeTensor because the ONNX model stores all initializers
When using torch.export.ExportedProgram as input, the initializers are
lifted as inputs. In order to execute the ONNX model, we need to pass a
reference to the non-fake model to the
ONNXProgram.adapt_torch_inputs_to_onnx API, so that initializers can be
fetched from the model and fed to the ONNX model as input
ps: #115461 will track the API revision for the cases where additional
model_with_state_dictare required to produce complete ONNX files exported with fake support. This is also tracked by the umbrella fake tensor issue #105464 FYI @BowenBao