-
Notifications
You must be signed in to change notification settings - Fork 25.7k
[ONNX] Support None in fx.args as torchlib inputs #108708
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
[ghstack-poisoned]
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/108708
Note: Links to docs will display an error until the docs builds have been completed. ✅ You can merge normally! (2 Unrelated Failures)As of commit c961589 with merge base bde75eb ( UNSTABLE - The following jobs failed but were likely due to flakiness present on trunk and has been marked as unstable:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
Previous to this PR, if None is returned from intermediate nodes, it will crashes the export because None is not expected to be passed into `_fill_tensor_shape_type`, and raise beartype roar. The function fills in shape and type to TorchScriptTensor according to its info from FX graph. This is discovered after microsoft/onnxscript#1043 is supported. The op specifically generates None in one of its inputs, but the only output from it being consumed is the first one (not None). Reference test from a TorchBench model: ```python def test_nanogpt(self): import sys sys.path.append("/home/titaiwang") from nanoGPT.model import GPT, GPTConfig # Load the model kwargs = { "block_size": 256, "vocab_size": 8096, # GPT-2 vocab_size of 50257, padded up to nearest multiple of 64 for efficiency "n_layer": 2, "n_head": 2, "n_embd": 128, "dropout": 0.0, "bias": False, # True: bias in Linears and LayerNorms, like GPT-2. False: a bit better and faster } config = GPTConfig(**kwargs) with torch.backends.cuda.sdp_kernel( enable_flash=True, enable_mem_efficient=True ): model = GPT(config) print("Done loading model") inputs = torch.arange(128).view(2, 64) targets = torch.arange(128).view(2, 64) self.run_test_with_fx_to_onnx_exporter_and_onnx_runtime( model, (inputs,), input_kwargs={ "targets": targets, }, verbose=True, ) ``` [ghstack-poisoned]
Do you mean an input or output is None? If output which one? |
| # TODO(titaiwang): set shape? | ||
| if isinstance(expected_value, (torch.SymInt, torch.SymFloat, torch.SymBool)): | ||
| if expected_value is None: | ||
| # There is no shape/type from None. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would link to the example you shown so readers know when this will happen
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would it be possible for use to assume it is always a scalar? Or could there be other cases?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hmmm it all depends on what fx.node gives us. This is pure product from fx graph. So I feel like what we should is only taking None into consideration of our code.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My guess is that for CPU, those outputs are useless or non-generated, so it returns None.
Prior to this PR, if None is returned from intermediate nodes, it will crashes the export because None is not expected to be passed into `_fill_tensor_shape_type`, and raise beartype roar. The function fills in shape and type to TorchScriptTensor according to its info from FX graph. This is discovered after microsoft/onnxscript#1043 is supported. The op specifically generates None in one of its inputs, but the only output from it being consumed is the first one (not None). Reference test from a TorchBench model: ```python def test_nanogpt(self): import sys sys.path.append("/home/titaiwang") from nanoGPT.model import GPT, GPTConfig # Load the model kwargs = { "block_size": 256, "vocab_size": 8096, # GPT-2 vocab_size of 50257, padded up to nearest multiple of 64 for efficiency "n_layer": 2, "n_head": 2, "n_embd": 128, "dropout": 0.0, "bias": False, # True: bias in Linears and LayerNorms, like GPT-2. False: a bit better and faster } config = GPTConfig(**kwargs) with torch.backends.cuda.sdp_kernel( enable_flash=True, enable_mem_efficient=True ): model = GPT(config) print("Done loading model") inputs = torch.arange(128).view(2, 64) targets = torch.arange(128).view(2, 64) self.run_test_with_fx_to_onnx_exporter_and_onnx_runtime( model, (inputs,), input_kwargs={ "targets": targets, }, verbose=True, ) ``` [ghstack-poisoned]
|
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Stack from ghstack (oldest at bottom):
Prior to this PR, if None is returned from intermediate nodes, it will crashes the export because None is not expected to be passed into
_fill_tensor_shape_type, and raise beartype roar. The function fills in shape and type to TorchScriptTensor according to its info from FX graph.This is discovered after microsoft/onnxscript#1043 is supported. The op specifically generates None in one of its inputs, but the only output from it being consumed is the first one (not None).
Reference test from a TorchBench model: